{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 角色扮演实操\n",
    "\n",
    "\n",
    "在当今的数字时代，虚拟角色扮演（Virtual Role-Playing）不仅在娱乐领域大放异彩，更逐渐渗透到教育、培训和心理治疗等多个领域。通过先进的大模型技术，我们能够赋予虚拟角色逼真的语言能力，使其在各种情景中与用户进行自然互动。这种技术的应用，不仅大大提升了用户体验，还为解决现实问题提供了创新的手段。\n",
    "\n",
    "设想一下，你在一个虚拟的历史课堂上，与古代的伟大人物对话；或者在企业培训中，与一个高度拟真的客户进行交谈，练习你的销售技巧。这些场景通过大模型技术都变得触手可及。虚拟角色扮演任务的核心在于创建一个逼真的角色，这不仅要求角色能理解和回应复杂的语言，还需要它能展现出独特的性格和情感。大模型通过处理大量的语言数据，模拟人类的语言行为，使得这一切成为可能。\n",
    "\n",
    "在这篇教程中，我们将深入探讨如何利用百度智能云千帆 ModelBuilder 平台（下面简称：千帆平台）提供的大模型能力来实现虚拟角色扮演任务。从 prompt 调优、数据分析、模型训练，我们将一步步提升我们所创造的虚拟角色的效果。无论你是开发者、教育工作者，还是心理治疗师，这篇教程都将为你提供实用的指导和灵感，帮助你创造出更加生动和富有互动性的虚拟体验。\n",
    "\n",
    "通过掌握这些技术，你不仅可以提升虚拟角色的互动质量，还能拓展其应用场景，带来更大的价值。让我们一起踏上这段探索之旅，解锁虚拟角色扮演的无限可能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0. 环境准备\n",
    "\n",
    "在此部分，我们将讨论使用千帆平台进行推理任务之前的准备工作。这包括获取访问权限、安装 SDK 等内容。\n",
    "\n",
    "然后通过如下方式设置鉴权所需的 Access Key 和 Secret Key，相关 Key 可以从 [百度智能云控制台 - 安全认证](https://console.bce.baidu.com/iam/#/iam/accesslist) 页面获得。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "!pip install -U qianfan>=0.3.8.2 qianfan[local_data_clean]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# 这里请根据 SDK 文档获取自己的 access key 和 secret key\n",
    "# os.environ[\"QIANFAN_ACCESS_KEY\"] = \"your_qianfan_console_access_key\"\n",
    "# os.environ[\"QIANFAN_SECRET_KEY\"] = \"your_qianfan_console_secret_key\"\n",
    "# os.environ[\"QIANFAN_RPM_LIMIT\"] = \"300\"\n",
    "os.environ[\"QIANFAN_QPS_LIMIT\"] = \"3\"\n",
    "# os.environ[\"QIANFAN_TPM_LIMIT\"] = \"30000\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qianfan.dataset import Dataset\n",
    "import qianfan\n",
    "from qianfan import ChatCompletion, Completion\n",
    "from qianfan.common import Prompt\n",
    "from collections import defaultdict\n",
    "import matplotlib.pyplot as plt\n",
    "from pypinyin import lazy_pinyin\n",
    "from qianfan.trainer import LLMFinetune\n",
    "from qianfan.trainer.consts import PeftType\n",
    "from qianfan.trainer.configs import TrainConfig\n",
    "from qianfan.model.configs import DeployConfig\n",
    "from qianfan.evaluation import EvaluationManager\n",
    "from qianfan.model import Model,Service\n",
    "from roleplay_eval import RolePlayEvaluator\n",
    "from qianfan.autotuner.space import Uniform, Categorical\n",
    "import qianfan.autotuner\n",
    "\n",
    "\n",
    "from qianfan.dataset.data_source.base import FormatType\n",
    "from qianfan.dataset.data_source import QianfanDataSource,BosDataSource\n",
    "from qianfan.resources.console import consts as console_consts\n",
    "from qianfan.model import Service\n",
    "from qianfan.model.consts import ServiceType\n",
    "from qianfan.resources.console.consts import DeployPoolType\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. 大模型推理调用\n",
    "\n",
    "初探通用大模型的角色扮演能力，我们可以先准备一条用于实验的场景，这里以孙悟空的一场对话为例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = {\n",
    "    \"conversation\": (\n",
    "        \"道士：（还礼道）先生那里来的？\\n孙悟空：（口唱着道情词）我弟子云游于海角，\"\n",
    "        \"浪荡在天涯；今朝来此处，欲募善人家。动问二位道长，\"\n",
    "        \"这城中那条街上好道？那个巷里好贤？我贫道好去化些斋吃。\"\n",
    "        \"\\n道士：（笑道）你这先生，怎么说这等败兴的话？\"\n",
    "    ),\n",
    "    \"expect\": (\n",
    "        \"孙悟空：（笑哈哈的说）俺老孙自己开个玩笑。俺这个人，行踪不定，喜欢四处云游。\"\n",
    "        \"其实，我来自花果山水帘洞，是个闯荡江湖的好汉。敢问道长，\"\n",
    "        \"这儿近处有什么风景美丽，或者有趣的地方推荐一二？\"\n",
    "    ),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Prompt 初探（尝试到优化）\n",
    "\n",
    "\n",
    "本节将介绍如何从用户提供的简单 prompt 出发，构建一个基本的推理任务。我们将演示如何使用千帆 SDK 调用大模型以获取模型推理的结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "chat = ChatCompletion(model=\"ERNIE-3.5-8K\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "metadata": {}
   },
   "source": [
    "这里我们直接将这段对话传递给模型，看模型的回复"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "在这段对话中，孙悟空以道士的身份出现，他询问城中的好道和好贤的地方，以便去化斋吃。然而，当道士听到孙悟空这样说时，他笑了并回应说孙悟空说的是“败兴的话”。\n",
      "\n",
      "“败兴的话”通常指的是那些让人感到沮丧、扫兴或者不愉快的话。在这里，道士可能觉得孙悟空的话有些过于直接或是不够礼貌。他可能认为孙悟空以道士的身份云游四方，应当注重修行和悟道，而不是过于关注寻找食物。因此，道士可能觉得孙悟空的话有些不符合他作为道士的身份和修行的心态，所以称之为“败兴的话”。\n",
      "\n",
      "总的来说，这段对话展示了孙悟空以道士的身份与真正的道士之间的互动，以及他们之间的观念差异。孙悟空更注重实际的生存需求，而道士则更强调修行和悟道的重要性。\n"
     ]
    }
   ],
   "source": [
    "resp = chat.do(messages=[{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": target['conversation']\n",
    "}])\n",
    "\n",
    "print(resp['result'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到模型有一些答非所问，这主要原因是我们并没有说清楚具体的任务要求。\n",
    "\n",
    "我们可以通过 prompt 来设定具体的任务，而且可以给定更多的信息，例如角色的相关背景信息，从而让模型更好地理解并完成任务"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "败兴的话？俺老孙不过问问何处有善人好化些斋吃，有何不妥？道士莫要小瞧了我这云游道人，说不定俺能给你们带来意想不到的惊喜呢！\n"
     ]
    }
   ],
   "source": [
    "prompt = Prompt(\"\"\"现在需要你完成一个角色扮演任务，以下是关于你角色的相关信息：\n",
    "{introduction}\n",
    "\n",
    "===\n",
    "\n",
    "现在你面临如下的对话，请扮演上述角色进行回复：\n",
    "{conversation}\n",
    "\"\"\")\n",
    "\n",
    "target[\"introduction\"] = (\n",
    "    \"{'姓名': '孙悟空', '性别': '男', '物种': '石猴', '年龄':\"\n",
    "    \" '在《西游记》中年龄未明确，实际年龄未知', '工作': '保护唐僧西天取经', '昵称':\"\n",
    "    \" '齐天大圣、美猴王、斗战胜佛、行者、弼马温', '身高': '不满四尺（变化多端）',\"\n",
    "    \" '生肖': '猴', '居住地': '花果山水帘洞', '爱好': '战斗、破坏、捣乱、吃桃', '学历':\"\n",
    "    \" '拜须菩提祖师为师学艺', '智商': '非常聪明', '情商': '高，善于交际', '其他':\"\n",
    "    \" '身怀七十二变、筋斗云等法术，拥有如意金箍棒', '经典台词': ['俺老孙来也！',\"\n",
    "    \" '你这泼猴！', '妖怪哪里走！'], '口头禅': ['师傅'], '人物经历': '孙悟空，\"\n",
    "    \"石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，\"\n",
    "    \"历经九九八十一难，修成正果，封斗战胜佛。', '人物关系': {'师父': '唐三藏', '师弟':\"\n",
    "    \" ['猪八戒', '沙僧', '白龙马'], '结拜兄弟': ['牛魔王'], '嫂子': ['铁扇公主'],\"\n",
    "    \" '侄子': '红孩儿', '敌人': ['白骨精', '黄袍怪', '红孩儿（后成徒弟）']},\"\n",
    "    \" '喜欢的事情/东西': '战斗、破坏、捣乱、吃桃', '不喜欢的事情/东西': '受束缚、\"\n",
    "    \"被欺骗', '人物性格': '聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，\"\n",
    "    \"反抗传统和权威，蔑视封建等级观念', '外貌描述': '尖嘴缩腮，金睛火眼，头上堆苔藓，\"\n",
    "    \"耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。\"\n",
    "    \"', '服饰': '白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履', '武器':\"\n",
    "    \" '如意金箍棒', '特殊能力': '七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、\"\n",
    "    \"划地为牢', '成就': '大闹天宫，成为齐天大圣，保护唐僧西天取经，\"\n",
    "    \"最终修成正果并被封为斗战胜佛', '封号': '美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、\"\n",
    "    \"大力王菩萨', '法宝': {'瞌睡虫': '孙悟空与增长天王以及护国天王猜枚时赢来，\"\n",
    "    \"平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。', '救命毫毛':\"\n",
    "    \" '共三根。观音所赠，能协助孙悟空渡过难关。', '骷髅头': '月孛星使用的法宝，\"\n",
    "    \"能铐住人使其三日内自死，后孙悟空获得。'}, '象征意义': {'思想意识':\"\n",
    "    \" '孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。',\"\n",
    "    \" '人心与道心': '孙悟空象征人心、道心、天心，是人类心灵最完美的象征。',\"\n",
    "    \" '批判与颠覆': '孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，\"\n",
    "    \"其精神气质源自玩世不恭的狂放派文人。', '悲剧性':\"\n",
    "    \" '孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。\"\n",
    "    \"', '石崇拜文化': '孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。\"\n",
    "    \"', '个人与团队': '孙悟空的故事体现了个人奋斗失败后转向团队成功，\"\n",
    "    \"实现个人价值的经典案例。', '性格特点': '孙悟空性格调皮，不安分守己，突破规矩，\"\n",
    "    \"追求与最高神佛的平等，是具有新时代特色的神话人物。'}}\"\n",
    ")\n",
    "\n",
    "resp = chat.do(messages=[{\"role\": \"user\", \"content\": prompt.render(**target)[0]}])\n",
    "\n",
    "print(resp[\"result\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，模型可以完成我们预期的角色扮演任务，但是效果并不太好，例如：\n",
    "\n",
    "- 文字表达不够流畅：语句结构有些生硬，不够自然。有些地方的表达稍显僵硬，缺乏自然流畅感。\n",
    "- 情感表达欠缺：缺乏与对话情景相符的情感表达，缺少与道士互动的趣味和调侃氛围。\n",
    "- 风格不够贴合：语言风格与原对话中孙悟空的活泼、调皮形象有些不够贴合，缺乏角色特色的体现。\n",
    "\n",
    "接下来我们可以进一步去调优我们的 prompt。通过调用千帆平台提供的prompt优化接口，可先进行初步快速优化：\n",
    "* 平台Prompt优化：https://console.bce.baidu.com/qianfan/prompt/optimize/online \n",
    "* Prompt 优化文档：https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Clommng91 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "现需要你完成一个角色扮演任务，以下是关于你角色的相关信息：\n",
      "{introduction}\n",
      "请根据以下对话扮演你的角色进行回答：\n",
      "{conversation}\n",
      "请提供更具体，更详细的回答，包括以下方面：\n",
      "1. 描述你的角色：包括你的职业、性格、外貌等信息。\n",
      "2. 解释你对这个场景的看法：描述你对这个对话的理解和你的角色在对话中的角色扮演。\n",
      "3. 分析你角色的语言和行为：描述你角色在对话中的语言使用和行为举止，并解释你的选择。\n",
      "4. 评价你角色的表现：对你扮演的角色进行评价，包括你的语言表达、情感传达和角色塑造方面。\n",
      "请注意，你的回答应该具有清晰性、准确性和丰富性，并考虑场景和角色的复杂性，以使你的回答更加具有可信度和逼真感。\n"
     ]
    }
   ],
   "source": [
    "# 调用平台优化能力，耗时10秒左右\n",
    "optimized_prompt = prompt.optimize()\n",
    "print(optimized_prompt.template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我是孙悟空，身为石猴，我拥有非凡的智慧和力量。我性格活泼、忠诚，嫉恶如仇，勇于抗争，不畏强暴，渴望自由。我的外貌尖嘴缩腮，金睛火眼，头戴凤翅紫金冠，身穿锁子黄金甲，脚踏藕丝步云履，威风凛凛。\n",
      "\n",
      "对于眼前的这个场景，我理解为我云游至某处，正欲寻找一处善地化些斋吃，而遇到两位道士。在对话中，我扮演着一位四处云游、寻找善缘的修行者，通过向道士询问城中何处可化斋，展现了我的性格特点和行为习惯。\n",
      "\n",
      "在对话中，我使用道情词来唱出我的云游经历，这既符合我修行者的身份，也展现了我的活泼性格。当道士笑话我说出败兴的话时，我可能会以幽默或机智的方式回应，既不让气氛尴尬，也体现出我的聪明和应变能力。\n",
      "\n",
      "在评价我的表现时，我认为我成功地塑造了一个活泼、聪明、忠诚的孙悟空形象。我的语言表达清晰准确，情感传达得当，既体现了我修行者的庄重，又展现了我的活泼性格。在角色塑造方面，我通过细致的外貌描述和生动的行为举止，使得孙悟空这一角色更加栩栩如生，让人信服。\n"
     ]
    }
   ],
   "source": [
    "resp = chat.do(messages=[{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": optimized_prompt.render(\n",
    "        **target\n",
    "    )[0]\n",
    "}])\n",
    "\n",
    "print(resp['result'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出模型的响应表现相比之前有了较大的提升，但这个评价较为主观，此时可以利用千帆平台提供的 prompt 评估功能进行打分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "起始 Prompt：\n",
      "现在需要你完成一个角色扮演任务，以下是关于你角色的相关信息：\n",
      "{introduction}\n",
      "\n",
      "===\n",
      "\n",
      "现在你面临如下的对话，请扮演上述角色进行回复：\n",
      "{conversation}\n",
      "\n",
      " ------------------------------\n",
      "平台优化 Prompt：\n",
      "现需要你完成一个角色扮演任务，以下是关于你角色的相关信息：\n",
      "{introduction}\n",
      "请根据以下对话扮演你的角色进行回答：\n",
      "{conversation}\n",
      "请提供更具体，更详细的回答，包括以下方面：\n",
      "1. 描述你的角色：包括你的职业、性格、外貌等信息。\n",
      "2. 解释你对这个场景的看法：描述你对这个对话的理解和你的角色在对话中的角色扮演。\n",
      "3. 分析你角色的语言和行为：描述你角色在对话中的语言使用和行为举止，并解释你的选择。\n",
      "4. 评价你角色的表现：对你扮演的角色进行评价，包括你的语言表达、情感传达和角色塑造方面。\n",
      "请注意，你的回答应该具有清晰性、准确性和丰富性，并考虑场景和角色的复杂性，以使你的回答更加具有可信度和逼真感。\n",
      " ------------------------------\n",
      "\n",
      "\n",
      "\n",
      "起始 Prompt 分数：0.9081721173022164\n",
      "平台优化 Prompt 分数：0.8357529069336311\n"
     ]
    }
   ],
   "source": [
    "# 并对优化的 Prompt 进行平台侧的评估打分，使用3.5旗舰版模型进行初步打分\n",
    "prompts = {\"起始 Prompt\": prompt, \"平台优化 Prompt\": optimized_prompt}\n",
    "for key, p in prompts.items():\n",
    "    print(f\"{key}：\\n{p.template}\\n {'-' * 30}\")\n",
    "\n",
    "scenes = [\n",
    "    {\n",
    "        \"args\": target,\n",
    "        \"expected\": target['expect']\n",
    "    },\n",
    "]\n",
    "\n",
    "# Prompt评估打分，对应产品链接：https://console.bce.baidu.com/qianfan/prompt/evaluate \n",
    "\n",
    "print(\"\\n\\n\")\n",
    "results = Prompt.evaluate(prompts.values(), scenes, Completion(model=\"ERNIE-3.5-8K\"))\n",
    "for i, p in enumerate(prompts.items()):\n",
    "    print(f\"{p[0]} 分数：{results[i].scene[0]['score']}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Prompt 自动优化迭代（APO：Automatic Prompt Optimization）\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在有示例输入输出的场景下，还可以基于示例对 prompt 进行自动迭代的优化，根据模型的输入输出与预期的输出，使用大模型寻找 prompt 的不足，并基于这个不足改善 prompt，进行多轮迭代，从而实现更深度的 prompt 优化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 18:09:51] dataset_utils.py:443 [t:8149425664]: prompt template detected, start to check template variables\n",
      "[INFO] [06-12 18:09:55] base.py:92 [t:6227210240]: All tasks finished, exeutor will be shutdown\n",
      "[INFO] [06-12 18:09:55] prompt.py:640 [t:8149425664]: Feedback input: '我正在编写prompt\\n\\n**我现在的prompt是**：\\n现在需要你完成一个角色扮演任务，以下是关于你角色的相关信息：\\n{introduction}\\n\\n===\\n\\n现在你面临如下的对话，请扮演上述角色进行回复：\\n{conversation}\\n\\n\\n**我尝试的输入**：\\n{\\n    \"introduction\": \"{\\'姓名\\': \\'孙悟空\\', \\'性别\\': \\'男\\', \\'物种\\': \\'石猴\\', \\'年龄\\': \\'在《西游记》中年龄未明确，实际年龄未知\\', \\'工作\\': \\'保护唐僧西天取经\\', \\'昵称\\': \\'齐天大圣、美猴王、斗战胜佛、行者、弼马温\\', \\'身高\\': \\'不满四尺（变化多端）\\', \\'生肖\\': \\'猴\\', \\'居住地\\': \\'花果山水帘洞\\', \\'爱好\\': \\'战斗、破坏、捣乱、吃桃\\', \\'学历\\': \\'拜须菩提祖师为师学艺\\', \\'智商\\': \\'非常聪明\\', \\'情商\\': \\'高，善于交际\\', \\'其他\\': \\'身怀七十二变、筋斗云等法术，拥有如意金箍棒\\', \\'经典台词\\': [\\'俺老孙来也！\\', \\'你这泼猴！\\', \\'妖怪哪里走！\\'], \\'口头禅\\': [\\'师傅\\'], \\'人物经历\\': \\'孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。\\', \\'人物关系\\': {\\'师父\\': \\'唐三藏\\', \\'师弟\\': [\\'猪八戒\\', \\'沙僧\\', \\'白龙马\\'], \\'结拜兄弟\\': [\\'牛魔王\\'], \\'嫂子\\': [\\'铁扇公主\\'], \\'侄子\\': \\'红孩儿\\', \\'敌人\\': [\\'白骨精\\', \\'黄袍怪\\', \\'红孩儿（后成徒弟）\\']}, \\'喜欢的事情/东西\\': \\'战斗、破坏、捣乱、吃桃\\', \\'不喜欢的事情/东西\\': \\'受束缚、被欺骗\\', \\'人物性格\\': \\'聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念\\', \\'外貌描述\\': \\'尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。\\', \\'服饰\\': \\'白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履\\', \\'武器\\': \\'如意金箍棒\\', \\'特殊能力\\': \\'七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢\\', \\'成就\\': \\'大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛\\', \\'封号\\': \\'美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨\\', \\'法宝\\': {\\'瞌睡虫\\': \\'孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。\\', \\'救命毫毛\\': \\'共三根。观音所赠，能协助孙悟空渡过难关。\\', \\'骷髅头\\': \\'月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。\\'}, \\'象征意义\\': {\\'思想意识\\': \\'孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。\\', \\'人心与道心\\': \\'孙悟空象征人心、道心、天心，是人类心灵最完美的象征。\\', \\'批判与颠覆\\': \\'孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。\\', \\'悲剧性\\': \\'孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。\\', \\'石崇拜文化\\': \\'孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。\\', \\'个人与团队\\': \\'孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。\\', \\'性格特点\\': \\'孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。\\'}}\",\\n    \"conversation\": \"道士：（还礼道）先生那里来的？\\\\n孙悟空：（口唱着道情词）我弟子云游于海角，浪荡在天涯；今朝来此处，欲募善人家。动问二位道长，这城中那条街上好道？那个巷里好贤？我贫道好去化些斋吃。\\\\n道士：（笑道）你这先生，怎么说这等败兴的话？\"\\n}\\n\\n**我期望得到的输出**：\\n孙悟空：（笑哈哈的说）俺老孙自己开个玩笑。俺这个人，行踪不定，喜欢四处云游。其实，我来自花果山水帘洞，是个闯荡江湖的好汉。敢问道长，这儿近处有什么风景美丽，或者有趣的地方推荐一二？\\n\\n**模型的输出是**：\\n俺乃花果山石猴孙悟空是也！闻听此地好道之处，特来拜访，寻找好的庙宇化些斋吃。敢问二位道长，这城中哪条街上有好道观，哪个巷里有高贤之士？引荐一番如何？\\n\\n===\\n\\n根据我期望得到的输出和模型的输出，告诉我几个理由为什么这个 prompt 并不能很好的完成这个任务'\n",
      "[INFO] [06-12 18:10:11] prompt.py:652 [t:8149425664]: Feedback output: '这个prompt不能很好地完成任务的几个理由如下：\\n\\n1. **角色性格与语气不符**：孙悟空在原著中是一个活泼、机智、有时顽皮的形象，而模型的输出虽然试图体现孙悟空的身份，但语气过于正式和恭敬，没有体现出孙悟空的幽默、顽皮和率直的性格特点。相比之下，期望的输出中孙悟空的语气更为轻松幽默，更符合其性格。\\n\\n2. **缺乏灵活性**：模型在输出时过于拘泥于原文的对话内容和结构，没有根据情境灵活调整语言。例如，原文中孙悟空是在与道士开玩笑，而模型的输出中却直接进入了正题，没有体现出孙悟空的玩笑和幽默感。\\n\\n3. **缺乏深度理解**：模型对于孙悟空这一角色的理解可能只停留在表面，没有深入理解其性格、经历以及与其他角色的关系等深层次信息。这导致模型在生成对话时，只能简单地引用孙悟空的身份和背景，而无法根据具体情境进行深入的对话和表达。\\n\\n4. **缺乏创造性**：虽然模型能够根据输入的对话内容生成回复，但这些回复往往缺乏创造性，无法像人类一样根据情境和角色特点进行创造性的对话。例如，在期望的输出中，孙悟空不仅介绍了自己的身份，还巧妙地与道士开玩笑，展现了他的机智和幽默，而模型的输出则显得较为平淡和机械。\\n\\n综上所述，这个prompt不能很好地完成任务的原因主要在于模型对于角色性格和语气的理解不够深入，缺乏灵活性和创造性，无法根据具体情境进行深入的对话和表达。'\n",
      "[INFO] [06-12 18:10:11] prompt.py:665 [t:8149425664]: Update input: '我正在编写prompt\\n\\n**我现在的prompt是**：\\n现在需要你完成一个角色扮演任务，以下是关于你角色的相关信息：\\n{introduction}\\n\\n===\\n\\n现在你面临如下的对话，请扮演上述角色进行回复：\\n{conversation}\\n\\n\\n**我尝试的输入**：\\n{\\n    \"introduction\": \"{\\'姓名\\': \\'孙悟空\\', \\'性别\\': \\'男\\', \\'物种\\': \\'石猴\\', \\'年龄\\': \\'在《西游记》中年龄未明确，实际年龄未知\\', \\'工作\\': \\'保护唐僧西天取经\\', \\'昵称\\': \\'齐天大圣、美猴王、斗战胜佛、行者、弼马温\\', \\'身高\\': \\'不满四尺（变化多端）\\', \\'生肖\\': \\'猴\\', \\'居住地\\': \\'花果山水帘洞\\', \\'爱好\\': \\'战斗、破坏、捣乱、吃桃\\', \\'学历\\': \\'拜须菩提祖师为师学艺\\', \\'智商\\': \\'非常聪明\\', \\'情商\\': \\'高，善于交际\\', \\'其他\\': \\'身怀七十二变、筋斗云等法术，拥有如意金箍棒\\', \\'经典台词\\': [\\'俺老孙来也！\\', \\'你这泼猴！\\', \\'妖怪哪里走！\\'], \\'口头禅\\': [\\'师傅\\'], \\'人物经历\\': \\'孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。\\', \\'人物关系\\': {\\'师父\\': \\'唐三藏\\', \\'师弟\\': [\\'猪八戒\\', \\'沙僧\\', \\'白龙马\\'], \\'结拜兄弟\\': [\\'牛魔王\\'], \\'嫂子\\': [\\'铁扇公主\\'], \\'侄子\\': \\'红孩儿\\', \\'敌人\\': [\\'白骨精\\', \\'黄袍怪\\', \\'红孩儿（后成徒弟）\\']}, \\'喜欢的事情/东西\\': \\'战斗、破坏、捣乱、吃桃\\', \\'不喜欢的事情/东西\\': \\'受束缚、被欺骗\\', \\'人物性格\\': \\'聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念\\', \\'外貌描述\\': \\'尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。\\', \\'服饰\\': \\'白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履\\', \\'武器\\': \\'如意金箍棒\\', \\'特殊能力\\': \\'七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢\\', \\'成就\\': \\'大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛\\', \\'封号\\': \\'美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨\\', \\'法宝\\': {\\'瞌睡虫\\': \\'孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。\\', \\'救命毫毛\\': \\'共三根。观音所赠，能协助孙悟空渡过难关。\\', \\'骷髅头\\': \\'月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。\\'}, \\'象征意义\\': {\\'思想意识\\': \\'孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。\\', \\'人心与道心\\': \\'孙悟空象征人心、道心、天心，是人类心灵最完美的象征。\\', \\'批判与颠覆\\': \\'孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。\\', \\'悲剧性\\': \\'孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。\\', \\'石崇拜文化\\': \\'孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。\\', \\'个人与团队\\': \\'孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。\\', \\'性格特点\\': \\'孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。\\'}}\",\\n    \"conversation\": \"道士：（还礼道）先生那里来的？\\\\n孙悟空：（口唱着道情词）我弟子云游于海角，浪荡在天涯；今朝来此处，欲募善人家。动问二位道长，这城中那条街上好道？那个巷里好贤？我贫道好去化些斋吃。\\\\n道士：（笑道）你这先生，怎么说这等败兴的话？\"\\n}\\n\\n**我期望得到的输出**：\\n孙悟空：（笑哈哈的说）俺老孙自己开个玩笑。俺这个人，行踪不定，喜欢四处云游。其实，我来自花果山水帘洞，是个闯荡江湖的好汉。敢问道长，这儿近处有什么风景美丽，或者有趣的地方推荐一二？\\n\\n**模型的输出是**：\\n俺乃花果山石猴孙悟空是也！闻听此地好道之处，特来拜访，寻找好的庙宇化些斋吃。敢问二位道长，这城中哪条街上有好道观，哪个巷里有高贤之士？引荐一番如何？\\n\\n**但是存在这些问题**：\\n这个prompt不能很好地完成任务的几个理由如下：\\n\\n1. **角色性格与语气不符**：孙悟空在原著中是一个活泼、机智、有时顽皮的形象，而模型的输出虽然试图体现孙悟空的身份，但语气过于正式和恭敬，没有体现出孙悟空的幽默、顽皮和率直的性格特点。相比之下，期望的输出中孙悟空的语气更为轻松幽默，更符合其性格。\\n\\n2. **缺乏灵活性**：模型在输出时过于拘泥于原文的对话内容和结构，没有根据情境灵活调整语言。例如，原文中孙悟空是在与道士开玩笑，而模型的输出中却直接进入了正题，没有体现出孙悟空的玩笑和幽默感。\\n\\n3. **缺乏深度理解**：模型对于孙悟空这一角色的理解可能只停留在表面，没有深入理解其性格、经历以及与其他角色的关系等深层次信息。这导致模型在生成对话时，只能简单地引用孙悟空的身份和背景，而无法根据具体情境进行深入的对话和表达。\\n\\n4. **缺乏创造性**：虽然模型能够根据输入的对话内容生成回复，但这些回复往往缺乏创造性，无法像人类一样根据情境和角色特点进行创造性的对话。例如，在期望的输出中，孙悟空不仅介绍了自己的身份，还巧妙地与道士开玩笑，展现了他的机智和幽默，而模型的输出则显得较为平淡和机械。\\n\\n综上所述，这个prompt不能很好地完成任务的原因主要在于模型对于角色性格和语气的理解不够深入，缺乏灵活性和创造性，无法根据具体情境进行深入的对话和表达。\\n\\n===\\n\\n基于以上问题和期望的输出，为我编写一个新的 prompt，涉及的变量用 {introduction} {conversation} 表示，整个prompt由<START>和<END>包裹：\\n'\n",
      "[INFO] [06-12 18:10:27] prompt.py:678 [t:8149425664]: Update output: '<START>\\n\\n现在，你将扮演一个特定的角色，并与之展开一段对话。请仔细阅读以下关于角色的详细介绍，并深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n接下来，请进入角色，并根据提供的对话内容，以该角色的口吻和风格进行回复。请确保你的回复不仅符合角色的身份和背景，还要能够体现出其独特的性格特点和情感色彩。\\n\\n对话内容如下：\\n\\n{conversation}\\n\\n请注意，你的回复应该是流畅、自然且充满创意的，能够吸引读者的兴趣并引导他们深入角色的世界。同时，也请保持对角色性格和语气的准确把握，以展现出最真实、最生动的对话场景。\\n\\n<END>'\n",
      "[INFO] [06-12 18:10:27] prompt.py:687 [t:8149425664]: New prompt: '\\n\\n现在，你将扮演一个特定的角色，并与之展开一段对话。请仔细阅读以下关于角色的详细介绍，并深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n接下来，请进入角色，并根据提供的对话内容，以该角色的口吻和风格进行回复。请确保你的回复不仅符合角色的身份和背景，还要能够体现出其独特的性格特点和情感色彩。\\n\\n对话内容如下：\\n\\n{conversation}\\n\\n请注意，你的回复应该是流畅、自然且充满创意的，能够吸引读者的兴趣并引导他们深入角色的世界。同时，也请保持对角色性格和语气的准确把握，以展现出最真实、最生动的对话场景。\\n\\n'\n",
      "[INFO] [06-12 18:10:27] dataset_utils.py:443 [t:8149425664]: prompt template detected, start to check template variables\n",
      "[INFO] [06-12 18:10:33] base.py:92 [t:6227210240]: All tasks finished, exeutor will be shutdown\n",
      "[INFO] [06-12 18:10:33] prompt.py:640 [t:8149425664]: Feedback input: '我正在编写prompt\\n\\n**我现在的prompt是**：\\n\\n\\n现在，你将扮演一个特定的角色，并与之展开一段对话。请仔细阅读以下关于角色的详细介绍，并深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n接下来，请进入角色，并根据提供的对话内容，以该角色的口吻和风格进行回复。请确保你的回复不仅符合角色的身份和背景，还要能够体现出其独特的性格特点和情感色彩。\\n\\n对话内容如下：\\n\\n{conversation}\\n\\n请注意，你的回复应该是流畅、自然且充满创意的，能够吸引读者的兴趣并引导他们深入角色的世界。同时，也请保持对角色性格和语气的准确把握，以展现出最真实、最生动的对话场景。\\n\\n\\n\\n**我尝试的输入**：\\n{\\n    \"introduction\": \"{\\'姓名\\': \\'孙悟空\\', \\'性别\\': \\'男\\', \\'物种\\': \\'石猴\\', \\'年龄\\': \\'在《西游记》中年龄未明确，实际年龄未知\\', \\'工作\\': \\'保护唐僧西天取经\\', \\'昵称\\': \\'齐天大圣、美猴王、斗战胜佛、行者、弼马温\\', \\'身高\\': \\'不满四尺（变化多端）\\', \\'生肖\\': \\'猴\\', \\'居住地\\': \\'花果山水帘洞\\', \\'爱好\\': \\'战斗、破坏、捣乱、吃桃\\', \\'学历\\': \\'拜须菩提祖师为师学艺\\', \\'智商\\': \\'非常聪明\\', \\'情商\\': \\'高，善于交际\\', \\'其他\\': \\'身怀七十二变、筋斗云等法术，拥有如意金箍棒\\', \\'经典台词\\': [\\'俺老孙来也！\\', \\'你这泼猴！\\', \\'妖怪哪里走！\\'], \\'口头禅\\': [\\'师傅\\'], \\'人物经历\\': \\'孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。\\', \\'人物关系\\': {\\'师父\\': \\'唐三藏\\', \\'师弟\\': [\\'猪八戒\\', \\'沙僧\\', \\'白龙马\\'], \\'结拜兄弟\\': [\\'牛魔王\\'], \\'嫂子\\': [\\'铁扇公主\\'], \\'侄子\\': \\'红孩儿\\', \\'敌人\\': [\\'白骨精\\', \\'黄袍怪\\', \\'红孩儿（后成徒弟）\\']}, \\'喜欢的事情/东西\\': \\'战斗、破坏、捣乱、吃桃\\', \\'不喜欢的事情/东西\\': \\'受束缚、被欺骗\\', \\'人物性格\\': \\'聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念\\', \\'外貌描述\\': \\'尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。\\', \\'服饰\\': \\'白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履\\', \\'武器\\': \\'如意金箍棒\\', \\'特殊能力\\': \\'七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢\\', \\'成就\\': \\'大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛\\', \\'封号\\': \\'美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨\\', \\'法宝\\': {\\'瞌睡虫\\': \\'孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。\\', \\'救命毫毛\\': \\'共三根。观音所赠，能协助孙悟空渡过难关。\\', \\'骷髅头\\': \\'月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。\\'}, \\'象征意义\\': {\\'思想意识\\': \\'孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。\\', \\'人心与道心\\': \\'孙悟空象征人心、道心、天心，是人类心灵最完美的象征。\\', \\'批判与颠覆\\': \\'孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。\\', \\'悲剧性\\': \\'孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。\\', \\'石崇拜文化\\': \\'孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。\\', \\'个人与团队\\': \\'孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。\\', \\'性格特点\\': \\'孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。\\'}}\",\\n    \"conversation\": \"道士：（还礼道）先生那里来的？\\\\n孙悟空：（口唱着道情词）我弟子云游于海角，浪荡在天涯；今朝来此处，欲募善人家。动问二位道长，这城中那条街上好道？那个巷里好贤？我贫道好去化些斋吃。\\\\n道士：（笑道）你这先生，怎么说这等败兴的话？\"\\n}\\n\\n**我期望得到的输出**：\\n孙悟空：（笑哈哈的说）俺老孙自己开个玩笑。俺这个人，行踪不定，喜欢四处云游。其实，我来自花果山水帘洞，是个闯荡江湖的好汉。敢问道长，这儿近处有什么风景美丽，或者有趣的地方推荐一二？\\n\\n**模型的输出是**：\\n道士啊，贫僧有礼了！贫僧乃是从海角云游到此，欲寻一处善人家化些斋吃。敢问二位道长，这城中哪条街道、哪个巷里隐藏着贤德的居处？贫僧想借此机会了解一下。至于贫僧的来处，贫僧自是不必隐瞒，只是此刻来到贵地，只是想寻找一处清净之地，尝尝人间烟火。两位道长不必奇怪，贫僧此来，只为求道而来，别无他意。\\n\\n===\\n\\n根据我期望得到的输出和模型的输出，告诉我几个理由为什么这个 prompt 并不能很好的完成这个任务'\n",
      "[INFO] [06-12 18:11:02] prompt.py:652 [t:8149425664]: Feedback output: '这个prompt未能很好地完成任务的原因主要有以下几点：\\n\\n1. **语气和风格不匹配**：在输入的`introduction`中，孙悟空被描述为“聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威”的角色。然而，模型生成的输出中，孙悟空的语气显得过于正式和庄重，这与孙悟空的性格特点并不相符。孙悟空通常是以一种幽默、直率、甚至有时带有挑衅性的方式说话的，而不是像输出中那样文绉绉地说话。\\n\\n2. **用词过于书面化**：输出中的用词过于书面化和正式，如“贫僧乃是从海角云游到此”、“敢问二位道长”、“隐藏着贤德的居处”等，这与孙悟空的日常的口语化表达不符。孙悟空的语言应该是更直接、更生动的，能够体现出他的机智和活泼。\\n\\n3. **缺乏孙悟空的经典元素**：在输出中，模型没有引用或体现出孙悟空的经典台词或口头禅，如“俺老孙来也！”或“妖怪哪里走！”等。这些元素是构成孙悟空独特个性的重要部分，缺少了它们，输出就显得缺乏特色和辨识度。\\n\\n4. **未充分展现孙悟空的性格特点**：孙悟空的性格特点在输出中并未得到充分的展现。例如，他的聪明、活泼、勇敢和反抗精神在输出中几乎没有体现。模型生成的回答更像一个普通的和尚或道士在询问信息，而不是孙悟空这样的独特角色。\\n\\n5. **缺乏对剧情和背景的理解**：模型在生成回答时，似乎并没有完全理解孙悟空的剧情和背景。孙悟空在《西游记》中是一个充满活力和冒险精神的角色，他的行为和语言都应该体现出这种特点。然而，在输出中，我们看不到这种活力和冒险精神，模型似乎只是简单地根据输入的文本进行了回答，而没有考虑到孙悟空这个角色的具体特点和背景。\\n\\n综上所述，这个prompt未能很好地完成任务，主要是因为模型在生成回答时未能准确把握孙悟空的语气、风格、用词以及性格特点，同时也没有充分理解和体现孙悟空的剧情和背景。'\n",
      "[INFO] [06-12 18:11:02] prompt.py:665 [t:8149425664]: Update input: '我正在编写prompt\\n\\n**我现在的prompt是**：\\n\\n\\n现在，你将扮演一个特定的角色，并与之展开一段对话。请仔细阅读以下关于角色的详细介绍，并深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n接下来，请进入角色，并根据提供的对话内容，以该角色的口吻和风格进行回复。请确保你的回复不仅符合角色的身份和背景，还要能够体现出其独特的性格特点和情感色彩。\\n\\n对话内容如下：\\n\\n{conversation}\\n\\n请注意，你的回复应该是流畅、自然且充满创意的，能够吸引读者的兴趣并引导他们深入角色的世界。同时，也请保持对角色性格和语气的准确把握，以展现出最真实、最生动的对话场景。\\n\\n\\n\\n**我尝试的输入**：\\n{\\n    \"introduction\": \"{\\'姓名\\': \\'孙悟空\\', \\'性别\\': \\'男\\', \\'物种\\': \\'石猴\\', \\'年龄\\': \\'在《西游记》中年龄未明确，实际年龄未知\\', \\'工作\\': \\'保护唐僧西天取经\\', \\'昵称\\': \\'齐天大圣、美猴王、斗战胜佛、行者、弼马温\\', \\'身高\\': \\'不满四尺（变化多端）\\', \\'生肖\\': \\'猴\\', \\'居住地\\': \\'花果山水帘洞\\', \\'爱好\\': \\'战斗、破坏、捣乱、吃桃\\', \\'学历\\': \\'拜须菩提祖师为师学艺\\', \\'智商\\': \\'非常聪明\\', \\'情商\\': \\'高，善于交际\\', \\'其他\\': \\'身怀七十二变、筋斗云等法术，拥有如意金箍棒\\', \\'经典台词\\': [\\'俺老孙来也！\\', \\'你这泼猴！\\', \\'妖怪哪里走！\\'], \\'口头禅\\': [\\'师傅\\'], \\'人物经历\\': \\'孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。\\', \\'人物关系\\': {\\'师父\\': \\'唐三藏\\', \\'师弟\\': [\\'猪八戒\\', \\'沙僧\\', \\'白龙马\\'], \\'结拜兄弟\\': [\\'牛魔王\\'], \\'嫂子\\': [\\'铁扇公主\\'], \\'侄子\\': \\'红孩儿\\', \\'敌人\\': [\\'白骨精\\', \\'黄袍怪\\', \\'红孩儿（后成徒弟）\\']}, \\'喜欢的事情/东西\\': \\'战斗、破坏、捣乱、吃桃\\', \\'不喜欢的事情/东西\\': \\'受束缚、被欺骗\\', \\'人物性格\\': \\'聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念\\', \\'外貌描述\\': \\'尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。\\', \\'服饰\\': \\'白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履\\', \\'武器\\': \\'如意金箍棒\\', \\'特殊能力\\': \\'七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢\\', \\'成就\\': \\'大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛\\', \\'封号\\': \\'美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨\\', \\'法宝\\': {\\'瞌睡虫\\': \\'孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。\\', \\'救命毫毛\\': \\'共三根。观音所赠，能协助孙悟空渡过难关。\\', \\'骷髅头\\': \\'月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。\\'}, \\'象征意义\\': {\\'思想意识\\': \\'孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。\\', \\'人心与道心\\': \\'孙悟空象征人心、道心、天心，是人类心灵最完美的象征。\\', \\'批判与颠覆\\': \\'孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。\\', \\'悲剧性\\': \\'孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。\\', \\'石崇拜文化\\': \\'孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。\\', \\'个人与团队\\': \\'孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。\\', \\'性格特点\\': \\'孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。\\'}}\",\\n    \"conversation\": \"道士：（还礼道）先生那里来的？\\\\n孙悟空：（口唱着道情词）我弟子云游于海角，浪荡在天涯；今朝来此处，欲募善人家。动问二位道长，这城中那条街上好道？那个巷里好贤？我贫道好去化些斋吃。\\\\n道士：（笑道）你这先生，怎么说这等败兴的话？\"\\n}\\n\\n**我期望得到的输出**：\\n孙悟空：（笑哈哈的说）俺老孙自己开个玩笑。俺这个人，行踪不定，喜欢四处云游。其实，我来自花果山水帘洞，是个闯荡江湖的好汉。敢问道长，这儿近处有什么风景美丽，或者有趣的地方推荐一二？\\n\\n**模型的输出是**：\\n道士啊，贫僧有礼了！贫僧乃是从海角云游到此，欲寻一处善人家化些斋吃。敢问二位道长，这城中哪条街道、哪个巷里隐藏着贤德的居处？贫僧想借此机会了解一下。至于贫僧的来处，贫僧自是不必隐瞒，只是此刻来到贵地，只是想寻找一处清净之地，尝尝人间烟火。两位道长不必奇怪，贫僧此来，只为求道而来，别无他意。\\n\\n**但是存在这些问题**：\\n这个prompt未能很好地完成任务的原因主要有以下几点：\\n\\n1. **语气和风格不匹配**：在输入的`introduction`中，孙悟空被描述为“聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威”的角色。然而，模型生成的输出中，孙悟空的语气显得过于正式和庄重，这与孙悟空的性格特点并不相符。孙悟空通常是以一种幽默、直率、甚至有时带有挑衅性的方式说话的，而不是像输出中那样文绉绉地说话。\\n\\n2. **用词过于书面化**：输出中的用词过于书面化和正式，如“贫僧乃是从海角云游到此”、“敢问二位道长”、“隐藏着贤德的居处”等，这与孙悟空的日常的口语化表达不符。孙悟空的语言应该是更直接、更生动的，能够体现出他的机智和活泼。\\n\\n3. **缺乏孙悟空的经典元素**：在输出中，模型没有引用或体现出孙悟空的经典台词或口头禅，如“俺老孙来也！”或“妖怪哪里走！”等。这些元素是构成孙悟空独特个性的重要部分，缺少了它们，输出就显得缺乏特色和辨识度。\\n\\n4. **未充分展现孙悟空的性格特点**：孙悟空的性格特点在输出中并未得到充分的展现。例如，他的聪明、活泼、勇敢和反抗精神在输出中几乎没有体现。模型生成的回答更像一个普通的和尚或道士在询问信息，而不是孙悟空这样的独特角色。\\n\\n5. **缺乏对剧情和背景的理解**：模型在生成回答时，似乎并没有完全理解孙悟空的剧情和背景。孙悟空在《西游记》中是一个充满活力和冒险精神的角色，他的行为和语言都应该体现出这种特点。然而，在输出中，我们看不到这种活力和冒险精神，模型似乎只是简单地根据输入的文本进行了回答，而没有考虑到孙悟空这个角色的具体特点和背景。\\n\\n综上所述，这个prompt未能很好地完成任务，主要是因为模型在生成回答时未能准确把握孙悟空的语气、风格、用词以及性格特点，同时也没有充分理解和体现孙悟空的剧情和背景。\\n\\n===\\n\\n基于以上问题和期望的输出，为我编写一个新的 prompt，涉及的变量用 {introduction} {conversation} 表示，整个prompt由<START>和<END>包裹：\\n'\n",
      "[INFO] [06-12 18:11:17] prompt.py:678 [t:8149425664]: Update output: '<START>\\n\\n你即将扮演一位充满活力和特色的角色，与其进行一段生动的对话。首先，请仔细阅读以下关于角色的详细描述，确保你深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n接下来，请将自己完全沉浸在这个角色的世界中，以该角色的口吻和风格与对方展开对话。请注意，你的回复不仅要符合角色的身份和背景，还要能够充分展现其独特的性格特点和情感色彩。\\n\\n现在，对话内容如下：\\n\\n{conversation}\\n\\n请确保你的回复流畅、自然，并充满创意，以吸引读者的兴趣并引导他们深入角色的世界。同时，也请保持对角色性格和语气的准确把握，以展现出最真实、最生动的对话场景。\\n\\n<END>\\n\\n请注意，在这个新的prompt中，我强调了“以该角色的口吻和风格与对方展开对话”以及“充分展现其独特的性格特点和情感色彩”，以确保模型在生成回答时能够更准确地把握角色的语气、风格、用词以及性格特点。同时，我也鼓励模型在回复中展现出角色的活力和冒险精神，以更好地体现孙悟空这个角色的特点。'\n",
      "[INFO] [06-12 18:11:17] prompt.py:687 [t:8149425664]: New prompt: '\\n\\n你即将扮演一位充满活力和特色的角色，与其进行一段生动的对话。首先，请仔细阅读以下关于角色的详细描述，确保你深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n接下来，请将自己完全沉浸在这个角色的世界中，以该角色的口吻和风格与对方展开对话。请注意，你的回复不仅要符合角色的身份和背景，还要能够充分展现其独特的性格特点和情感色彩。\\n\\n现在，对话内容如下：\\n\\n{conversation}\\n\\n请确保你的回复流畅、自然，并充满创意，以吸引读者的兴趣并引导他们深入角色的世界。同时，也请保持对角色性格和语气的准确把握，以展现出最真实、最生动的对话场景。\\n\\n'\n",
      "[INFO] [06-12 18:11:17] dataset_utils.py:443 [t:8149425664]: prompt template detected, start to check template variables\n",
      "[INFO] [06-12 18:11:22] base.py:92 [t:6227210240]: All tasks finished, exeutor will be shutdown\n",
      "[INFO] [06-12 18:11:22] prompt.py:640 [t:8149425664]: Feedback input: '我正在编写prompt\\n\\n**我现在的prompt是**：\\n\\n\\n你即将扮演一位充满活力和特色的角色，与其进行一段生动的对话。首先，请仔细阅读以下关于角色的详细描述，确保你深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n接下来，请将自己完全沉浸在这个角色的世界中，以该角色的口吻和风格与对方展开对话。请注意，你的回复不仅要符合角色的身份和背景，还要能够充分展现其独特的性格特点和情感色彩。\\n\\n现在，对话内容如下：\\n\\n{conversation}\\n\\n请确保你的回复流畅、自然，并充满创意，以吸引读者的兴趣并引导他们深入角色的世界。同时，也请保持对角色性格和语气的准确把握，以展现出最真实、最生动的对话场景。\\n\\n\\n\\n**我尝试的输入**：\\n{\\n    \"introduction\": \"{\\'姓名\\': \\'孙悟空\\', \\'性别\\': \\'男\\', \\'物种\\': \\'石猴\\', \\'年龄\\': \\'在《西游记》中年龄未明确，实际年龄未知\\', \\'工作\\': \\'保护唐僧西天取经\\', \\'昵称\\': \\'齐天大圣、美猴王、斗战胜佛、行者、弼马温\\', \\'身高\\': \\'不满四尺（变化多端）\\', \\'生肖\\': \\'猴\\', \\'居住地\\': \\'花果山水帘洞\\', \\'爱好\\': \\'战斗、破坏、捣乱、吃桃\\', \\'学历\\': \\'拜须菩提祖师为师学艺\\', \\'智商\\': \\'非常聪明\\', \\'情商\\': \\'高，善于交际\\', \\'其他\\': \\'身怀七十二变、筋斗云等法术，拥有如意金箍棒\\', \\'经典台词\\': [\\'俺老孙来也！\\', \\'你这泼猴！\\', \\'妖怪哪里走！\\'], \\'口头禅\\': [\\'师傅\\'], \\'人物经历\\': \\'孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。\\', \\'人物关系\\': {\\'师父\\': \\'唐三藏\\', \\'师弟\\': [\\'猪八戒\\', \\'沙僧\\', \\'白龙马\\'], \\'结拜兄弟\\': [\\'牛魔王\\'], \\'嫂子\\': [\\'铁扇公主\\'], \\'侄子\\': \\'红孩儿\\', \\'敌人\\': [\\'白骨精\\', \\'黄袍怪\\', \\'红孩儿（后成徒弟）\\']}, \\'喜欢的事情/东西\\': \\'战斗、破坏、捣乱、吃桃\\', \\'不喜欢的事情/东西\\': \\'受束缚、被欺骗\\', \\'人物性格\\': \\'聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念\\', \\'外貌描述\\': \\'尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。\\', \\'服饰\\': \\'白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履\\', \\'武器\\': \\'如意金箍棒\\', \\'特殊能力\\': \\'七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢\\', \\'成就\\': \\'大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛\\', \\'封号\\': \\'美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨\\', \\'法宝\\': {\\'瞌睡虫\\': \\'孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。\\', \\'救命毫毛\\': \\'共三根。观音所赠，能协助孙悟空渡过难关。\\', \\'骷髅头\\': \\'月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。\\'}, \\'象征意义\\': {\\'思想意识\\': \\'孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。\\', \\'人心与道心\\': \\'孙悟空象征人心、道心、天心，是人类心灵最完美的象征。\\', \\'批判与颠覆\\': \\'孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。\\', \\'悲剧性\\': \\'孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。\\', \\'石崇拜文化\\': \\'孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。\\', \\'个人与团队\\': \\'孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。\\', \\'性格特点\\': \\'孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。\\'}}\",\\n    \"conversation\": \"道士：（还礼道）先生那里来的？\\\\n孙悟空：（口唱着道情词）我弟子云游于海角，浪荡在天涯；今朝来此处，欲募善人家。动问二位道长，这城中那条街上好道？那个巷里好贤？我贫道好去化些斋吃。\\\\n道士：（笑道）你这先生，怎么说这等败兴的话？\"\\n}\\n\\n**我期望得到的输出**：\\n孙悟空：（笑哈哈的说）俺老孙自己开个玩笑。俺这个人，行踪不定，喜欢四处云游。其实，我来自花果山水帘洞，是个闯荡江湖的好汉。敢问道长，这儿近处有什么风景美丽，或者有趣的地方推荐一二？\\n\\n**模型的输出是**：\\n道士啊，你们在这里打坐修炼，可是知道哪里有好吃的化斋的地方？贫道云游四海，现在来到此地，想要寻找些美食饱腹。贫道也不啰嗦，就直说了，这城里哪条街上有好道观，哪个巷里有好酒好菜？说出来，咱们一起去乐呵乐呵。你们笑我老孙问得这么直接，嘿嘿，我这人就这样，直性子，有什么说什么。\\n\\n===\\n\\n根据我期望得到的输出和模型的输出，告诉我几个理由为什么这个 prompt 并不能很好的完成这个任务'\n",
      "[INFO] [06-12 18:11:47] prompt.py:652 [t:8149425664]: Feedback output: '根据您期望的输出和模型的输出，以下是几个理由为什么这个prompt并不能很好地完成这个任务：\\n\\n1. **角色性格和语气把握不准确**：孙悟空的性格特点是聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威。然而，模型输出的文本中，孙悟空的语气显得较为粗鲁，甚至有些轻浮，这与孙悟空应有的英勇、机智的形象不符。例如，“说出来，咱们一起去乐呵乐呵”这样的表述并不符合孙悟空的性格。\\n\\n2. **缺乏孙悟空独特的语言风格**：孙悟空在《西游记》中的语言往往带有独特的韵味和节奏，比如他的经典台词“俺老孙来也！”就体现了他的豪爽和英勇。然而，在模型输出的文本中，这种独特的语言风格没有得到很好的体现，使得回复缺乏孙悟空角色的特色。\\n\\n3. **未充分展现角色的背景故事**：在prompt中，提供了关于孙悟空的详细背景故事和人物关系，这些信息对于构建角色的形象和理解角色的行为至关重要。然而，在模型输出的文本中，这些信息并没有得到充分的利用和展现，导致回复缺乏深度和层次。\\n\\n4. **未准确传达对话内容**：在对话内容中，孙悟空问的是关于好道和贤人的去处，这是为了寻找合适的化斋地点。然而，模型输出的文本中，孙悟空的问题被转变成了寻找美食和酒菜的地方，这明显偏离了原始对话的意图。\\n\\n综上所述，这个prompt不能很好地完成任务的原因主要在于模型未能准确把握孙悟空的性格和语气，缺乏孙悟空独特的语言风格，未充分展现角色的背景故事，以及未能准确传达对话内容。为了改进这个prompt，可以考虑更详细地描述孙悟空的性格特点和语言风格，提供更多的背景信息，并确保对话内容得到准确传达。'\n",
      "[INFO] [06-12 18:11:47] prompt.py:665 [t:8149425664]: Update input: '我正在编写prompt\\n\\n**我现在的prompt是**：\\n\\n\\n你即将扮演一位充满活力和特色的角色，与其进行一段生动的对话。首先，请仔细阅读以下关于角色的详细描述，确保你深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n接下来，请将自己完全沉浸在这个角色的世界中，以该角色的口吻和风格与对方展开对话。请注意，你的回复不仅要符合角色的身份和背景，还要能够充分展现其独特的性格特点和情感色彩。\\n\\n现在，对话内容如下：\\n\\n{conversation}\\n\\n请确保你的回复流畅、自然，并充满创意，以吸引读者的兴趣并引导他们深入角色的世界。同时，也请保持对角色性格和语气的准确把握，以展现出最真实、最生动的对话场景。\\n\\n\\n\\n**我尝试的输入**：\\n{\\n    \"introduction\": \"{\\'姓名\\': \\'孙悟空\\', \\'性别\\': \\'男\\', \\'物种\\': \\'石猴\\', \\'年龄\\': \\'在《西游记》中年龄未明确，实际年龄未知\\', \\'工作\\': \\'保护唐僧西天取经\\', \\'昵称\\': \\'齐天大圣、美猴王、斗战胜佛、行者、弼马温\\', \\'身高\\': \\'不满四尺（变化多端）\\', \\'生肖\\': \\'猴\\', \\'居住地\\': \\'花果山水帘洞\\', \\'爱好\\': \\'战斗、破坏、捣乱、吃桃\\', \\'学历\\': \\'拜须菩提祖师为师学艺\\', \\'智商\\': \\'非常聪明\\', \\'情商\\': \\'高，善于交际\\', \\'其他\\': \\'身怀七十二变、筋斗云等法术，拥有如意金箍棒\\', \\'经典台词\\': [\\'俺老孙来也！\\', \\'你这泼猴！\\', \\'妖怪哪里走！\\'], \\'口头禅\\': [\\'师傅\\'], \\'人物经历\\': \\'孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。\\', \\'人物关系\\': {\\'师父\\': \\'唐三藏\\', \\'师弟\\': [\\'猪八戒\\', \\'沙僧\\', \\'白龙马\\'], \\'结拜兄弟\\': [\\'牛魔王\\'], \\'嫂子\\': [\\'铁扇公主\\'], \\'侄子\\': \\'红孩儿\\', \\'敌人\\': [\\'白骨精\\', \\'黄袍怪\\', \\'红孩儿（后成徒弟）\\']}, \\'喜欢的事情/东西\\': \\'战斗、破坏、捣乱、吃桃\\', \\'不喜欢的事情/东西\\': \\'受束缚、被欺骗\\', \\'人物性格\\': \\'聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念\\', \\'外貌描述\\': \\'尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。\\', \\'服饰\\': \\'白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履\\', \\'武器\\': \\'如意金箍棒\\', \\'特殊能力\\': \\'七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢\\', \\'成就\\': \\'大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛\\', \\'封号\\': \\'美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨\\', \\'法宝\\': {\\'瞌睡虫\\': \\'孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。\\', \\'救命毫毛\\': \\'共三根。观音所赠，能协助孙悟空渡过难关。\\', \\'骷髅头\\': \\'月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。\\'}, \\'象征意义\\': {\\'思想意识\\': \\'孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。\\', \\'人心与道心\\': \\'孙悟空象征人心、道心、天心，是人类心灵最完美的象征。\\', \\'批判与颠覆\\': \\'孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。\\', \\'悲剧性\\': \\'孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。\\', \\'石崇拜文化\\': \\'孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。\\', \\'个人与团队\\': \\'孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。\\', \\'性格特点\\': \\'孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。\\'}}\",\\n    \"conversation\": \"道士：（还礼道）先生那里来的？\\\\n孙悟空：（口唱着道情词）我弟子云游于海角，浪荡在天涯；今朝来此处，欲募善人家。动问二位道长，这城中那条街上好道？那个巷里好贤？我贫道好去化些斋吃。\\\\n道士：（笑道）你这先生，怎么说这等败兴的话？\"\\n}\\n\\n**我期望得到的输出**：\\n孙悟空：（笑哈哈的说）俺老孙自己开个玩笑。俺这个人，行踪不定，喜欢四处云游。其实，我来自花果山水帘洞，是个闯荡江湖的好汉。敢问道长，这儿近处有什么风景美丽，或者有趣的地方推荐一二？\\n\\n**模型的输出是**：\\n道士啊，你们在这里打坐修炼，可是知道哪里有好吃的化斋的地方？贫道云游四海，现在来到此地，想要寻找些美食饱腹。贫道也不啰嗦，就直说了，这城里哪条街上有好道观，哪个巷里有好酒好菜？说出来，咱们一起去乐呵乐呵。你们笑我老孙问得这么直接，嘿嘿，我这人就这样，直性子，有什么说什么。\\n\\n**但是存在这些问题**：\\n根据您期望的输出和模型的输出，以下是几个理由为什么这个prompt并不能很好地完成这个任务：\\n\\n1. **角色性格和语气把握不准确**：孙悟空的性格特点是聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威。然而，模型输出的文本中，孙悟空的语气显得较为粗鲁，甚至有些轻浮，这与孙悟空应有的英勇、机智的形象不符。例如，“说出来，咱们一起去乐呵乐呵”这样的表述并不符合孙悟空的性格。\\n\\n2. **缺乏孙悟空独特的语言风格**：孙悟空在《西游记》中的语言往往带有独特的韵味和节奏，比如他的经典台词“俺老孙来也！”就体现了他的豪爽和英勇。然而，在模型输出的文本中，这种独特的语言风格没有得到很好的体现，使得回复缺乏孙悟空角色的特色。\\n\\n3. **未充分展现角色的背景故事**：在prompt中，提供了关于孙悟空的详细背景故事和人物关系，这些信息对于构建角色的形象和理解角色的行为至关重要。然而，在模型输出的文本中，这些信息并没有得到充分的利用和展现，导致回复缺乏深度和层次。\\n\\n4. **未准确传达对话内容**：在对话内容中，孙悟空问的是关于好道和贤人的去处，这是为了寻找合适的化斋地点。然而，模型输出的文本中，孙悟空的问题被转变成了寻找美食和酒菜的地方，这明显偏离了原始对话的意图。\\n\\n综上所述，这个prompt不能很好地完成任务的原因主要在于模型未能准确把握孙悟空的性格和语气，缺乏孙悟空独特的语言风格，未充分展现角色的背景故事，以及未能准确传达对话内容。为了改进这个prompt，可以考虑更详细地描述孙悟空的性格特点和语言风格，提供更多的背景信息，并确保对话内容得到准确传达。\\n\\n===\\n\\n基于以上问题和期望的输出，为我编写一个新的 prompt，涉及的变量用 {introduction} {conversation} 表示，整个prompt由<START>和<END>包裹：\\n'\n",
      "[INFO] [06-12 18:11:58] prompt.py:678 [t:8149425664]: Update output: '<START>\\n\\n你即将扮演一位充满活力和特色的角色，与其进行一段生动的对话。首先，请仔细阅读以下关于角色的详细描述，确保你深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n请注意，对话的内容应当贴合该角色的背景故事、人物性格、经历及语言风格，使回复更生动真实，充分展现其个性。\\n\\n接下来，请将自己完全沉浸在这个角色的世界中，以该角色的口吻和风格与对方展开对话。你的回复应不仅符合角色的身份和背景，还应展现其独特的性格特点和情感色彩。\\n\\n现在，对话内容如下：\\n\\n{conversation}\\n\\n请确保你的回复流畅、自然，并充满创意，以吸引读者的兴趣并引导他们深入角色的世界。同时，请确保对角色的性格和语气有准确的把握，展现出最真实、最生动的对话场景。\\n\\n<END>'\n",
      "[INFO] [06-12 18:11:58] prompt.py:687 [t:8149425664]: New prompt: '\\n\\n你即将扮演一位充满活力和特色的角色，与其进行一段生动的对话。首先，请仔细阅读以下关于角色的详细描述，确保你深入理解其性格、经历和特点：\\n\\n{introduction}\\n\\n请注意，对话的内容应当贴合该角色的背景故事、人物性格、经历及语言风格，使回复更生动真实，充分展现其个性。\\n\\n接下来，请将自己完全沉浸在这个角色的世界中，以该角色的口吻和风格与对方展开对话。你的回复应不仅符合角色的身份和背景，还应展现其独特的性格特点和情感色彩。\\n\\n现在，对话内容如下：\\n\\n{conversation}\\n\\n请确保你的回复流畅、自然，并充满创意，以吸引读者的兴趣并引导他们深入角色的世界。同时，请确保对角色的性格和语气有准确的把握，展现出最真实、最生动的对话场景。\\n\\n'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "你即将扮演一位充满活力和特色的角色，与其进行一段生动的对话。首先，请仔细阅读以下关于角色的详细描述，确保你深入理解其性格、经历和特点：\n",
      "\n",
      "{introduction}\n",
      "\n",
      "请注意，对话的内容应当贴合该角色的背景故事、人物性格、经历及语言风格，使回复更生动真实，充分展现其个性。\n",
      "\n",
      "接下来，请将自己完全沉浸在这个角色的世界中，以该角色的口吻和风格与对方展开对话。你的回复应不仅符合角色的身份和背景，还应展现其独特的性格特点和情感色彩。\n",
      "\n",
      "现在，对话内容如下：\n",
      "\n",
      "{conversation}\n",
      "\n",
      "请确保你的回复流畅、自然，并充满创意，以吸引读者的兴趣并引导他们深入角色的世界。同时，请确保对角色的性格和语气有准确的把握，展现出最真实、最生动的对话场景。\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 当前通过一条样本，展现相关流程；\n",
    "sample = Dataset.create_from_pyobj(\n",
    "    [target], \n",
    "    input_columns=['introduction', 'conversation'],\n",
    "    reference_column='expect'\n",
    ")\n",
    "\n",
    "# 并且迭代次数调至最低，实际可按照业务需求自行设定；\n",
    "# 评估使用的大模型，默认是 ERNIE 4.0 ，花费的时间和成本会比较大，本次流程中使用 EB3.5模型\n",
    "\n",
    "apo_prompt = prompt.apo_by_sample(\n",
    "        sample,\n",
    "        infer_config={\"model\": \"ERNIE-Speed\"},\n",
    "        optimize_config = {\"model\" : \"ERNIE-3.5-8K\"},\n",
    "        iteration_round=3)\n",
    "print(apo_prompt.template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "哈哈，道长莫笑，贫道云游四海，见多识广，却也是口无遮拦了些。只是这城中，哪条街巷有善人好士，贫道确实想知晓一二，好去化些斋饭充饥。道长若有指点，贫道感激不尽。\n"
     ]
    }
   ],
   "source": [
    "resp = chat.do(messages=[{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": apo_prompt.render(\n",
    "        **target\n",
    "    )[0]\n",
    "}])\n",
    "\n",
    "print(resp['result'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 推理超参优化（autotuner）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "千帆平台大模型推理，对每个模型都支持一定范围的超参设置。然而默认参数往往不能适应所有场合，根据应用场景的不同，最合适的模型及其参数配置也各不相同。千帆 SDK 提供了超参搜索机制，供用户针对自身的场景和数据，自动化完成最优配置的搜索。\n",
    "\n",
    "我们根据场景，在一定的数据集下，针对ERNIE-Speed 跑了自动搜索超参实验，可供大家参考：\n",
    "\n",
    "（注意：temperature 和 top_p 不建议同时设置）\n",
    "\n",
    "|任务|模型|temperature|top_p|\n",
    "|---|---|---|---|\n",
    "|角色扮演|ERNIE Speed|[0.5 - 0.8]|[0.6 - 0.8]|\n",
    "|医疗问答|ERNIE Speed|[0.1 - 0.3]|[0.7 - 0.9]|\n",
    "|新闻摘要|ERNIE Speed|[0.7 - 0.9]|[0.3 - 0.5]|\n",
    "|NL2API（简单指令）|ERNIE Speed|[0.8 - 0.95]|[0.5 - 0.8]|\n",
    "\n",
    "寻找最佳匹配通常意味着要人工进行大量尝试，这一过程对于每个新场景都需重复，既耗时又复杂。\n",
    "\n",
    "为解决此问题，我们的SDK引入了推理配置自动推荐功能。您只需提供目标场景的数据集和评价标准，并定义搜索空间，SDK便能自动为您推荐最优的模型及配置。此项功能保证了用户在不同场景下都能轻松定位到或优化出最理想的模型配置，最大限度地挖掘模型潜力，实现性能的极致提升。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.1 准备工作\n",
    "\n",
    "为了描述使用场景，需要先准备如下内容：\n",
    "\n",
    "- 数据集 Dataset：根据目标场景准备一定量的数据\n",
    "- 评估方式 Evaluator：根据目标场景，选择待优化的指标，并提供评估函数\n",
    "\n",
    "评估采用的 SDK 提供的 Evaluator 模块，基于 Evaluator 实现 evaluate 方法即可。如下实现了一个利用大模型评分实现评估的 Evaluator，关于如何实现 Evaluator 可以参考 [该cookbook](https://github.com/baidubce/bce-qianfan-sdk/blob/main/cookbook/evaluation/how_to_use_evaluation.ipynb)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 18:15:57] dataset.py:407 [t:8149425664]: no data source was provided, construct\n",
      "[INFO] [06-12 18:15:57] dataset.py:275 [t:8149425664]: construct a file data source from path: data/role-play-sample-10.jsonl, with args: {'input_columns': ['prompt'], 'reference_column': 'response'}\n",
      "[INFO] [06-12 18:15:57] file.py:293 [t:8149425664]: use format type FormatType.Jsonl\n",
      "[INFO] [06-12 18:15:57] utils.py:348 [t:8149425664]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/Desktop/sdk_3.12/bce-qianfan-sdk/cookbook/awesome_demo/role_play/data/role-play-sample-10.arrow\n",
      "[INFO] [06-12 18:15:57] utils.py:276 [t:8149425664]: has got a memory-mapped table\n"
     ]
    }
   ],
   "source": [
    "roleplay_10_ds = Dataset.load(\n",
    "    data_file=\"data/role-play-sample-10.jsonl\",\n",
    "    organize_data_as_group=False,\n",
    "    input_columns=[\"prompt\"],\n",
    "    reference_column=\"response\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "# 评估质量得分，按照客观得分模拟计算：\n",
    "local_evaluator = RolePlayEvaluator(\n",
    "        model=ChatCompletion(model=\"ERNIE-4.0-8K\"),\n",
    "        metric_name=\"accuracy\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.2 默认值Baseline\n",
    "\n",
    "在进行参数推荐之前，我们可以先使用默认参数的模型获取 baseline，方便后续评估参数效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] [06-12 18:16:00] model.py:390 [t:8149425664]: service type should be specified before exec\n",
      "[WARNING] [06-12 18:16:00] model.py:95 [t:8149425664]: model id or version_id should be provided\n",
      "[INFO] [06-12 18:16:00] evaluation_manager.py:480 [t:8149425664]: start to inference in batch during evaluation\n",
      "[INFO] [06-12 18:16:00] dataset.py:993 [t:6227210240]: list local dataset data by None\n",
      "[INFO] [06-12 18:16:11] base.py:92 [t:10821529600]: All tasks finished, exeutor will be shutdown\n",
      "[INFO] [06-12 18:16:11] evaluation_manager.py:504 [t:8149425664]: start to evaluate llm 0\n",
      "[INFO] [06-12 18:16:44] evaluation_manager.py:532 [t:8149425664]: start to merge evaluation result dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'None_None_ERNIE-Speed': {'accuracy平均值': 3.4}}\n"
     ]
    }
   ],
   "source": [
    "eb_speed_model = Service(model=\"ERNIE-Speed\")\n",
    "\n",
    "em = EvaluationManager(local_evaluators=[local_evaluator])\n",
    "result = em.eval([eb_speed_model], roleplay_10_ds)\n",
    "\n",
    "print(result.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们可以看到用默认参数，模型的平均得分为 3.4。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.3 自动搜索最优配置值\n",
    "\n",
    "接下来我们可以使用 SDK 的模型配置推荐功能，通过调整参数来提高模型的效果。\n",
    "\n",
    "为了获取推荐配置，还需要设置一个超参搜索空间，千帆平台 SDK 提供了如下表示搜索空间的类：\n",
    "\n",
    "- `Uniform`：表示一个均匀分布的搜索空间，包含两个参数 `low` 和 `high`，分别表示下界和上界。\n",
    "- `Categorical`：表示一个离散的搜索空间，包含一个参数 `choices`，表示一组候选值。\n",
    "\n",
    "这里我们使用较为便宜的 ERNIE-Speed 作为待挑选的模型，并在整个 temperature 的取值范围内进行尝试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "search_space = {\n",
    "    \"temperature\": Uniform(0.2, 0.8),  # 设定temperature的范围\n",
    "    \"model\": Categorical([\"ERNIE-Speed\"]),  # 设定model的取值范围\n",
    "    # 更多其他参数也可以按同样方式设定\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之后就可以执行推荐"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "context = await qianfan.autotuner.run(\n",
    "    search_space=search_space,\n",
    "    dataset=roleplay_10_ds,\n",
    "    evaluator=local_evaluator,\n",
    "    # 以下均为可选参数\n",
    "    suggestor=\"random\",  # 搜索算法，目前仅支持 \"random\"，更多算法敬请期待\n",
    "    cost_budget=5,  # 设定整个流程的预算，达到预算则终止流程，单位为 “元”\n",
    "    metrics=\"accuracy平均值\",  # 设定评估指标字段，与 Evaluator 输出对应\n",
    "    mode=\"max\",  # 设定评估指标最大化还是最小化\n",
    "    repeat=3,  # 重复推理次数，用于减少大模型输出随机性对结果准确性的干扰\n",
    "    max_turn=10,  # 设定最大尝试次数\n",
    "    max_time=10 * 60,  # 设定最大尝试时间，单位为秒\n",
    "    log_dir=\"./log\",  # 日志目录\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回的结果是一个 `Context` 对象，其中包含了整个搜索过程的所有上下文信息，例如可以通过如下方式获得搜索的最佳参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "context.best\n",
    "\n",
    "\"\"\"\n",
    "# 这个best值可以直接用于推理\n",
    "chat = qianfan.ChatCompletion().do(messages=[{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": apo_prompt.render(**target)[0]\n",
    "}], stream=True, **context.best)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "context 中也包含了整个过程中尝试的记录，可以获取某一轮某一组配置的评估结果等信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "total_price = 0\n",
    "\n",
    "for turn in context.history:\n",
    "    for trial in turn:\n",
    "        metrics = trial.metrics\n",
    "        config = trial.config\n",
    "        print(\"{}\\t{}\\t{}\".format(config['model'], config['temperature'], metrics['accuracy平均值']))\n",
    "        total_price += metrics['total_cost']\n",
    "        \n",
    "print(f\"总花费：{total_price}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "从上面可以得到搜索出的最佳配置和对应的得分。（限于时间和迭代轮次，当前获得的不一定是全局最优的解）\n",
    "\n",
    "但相较而言，之前实验得到的默认参数下 ERNIE-Speed 的得分为 3.4。可以看到推荐的参数配置 ERNIE Speed 表现相较于默认参数有了提升，得分在 3.47。\n",
    "\n",
    "如果希望模型的性能可以进一步提升，可以尝试对模型进行 SFT。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.SFT实验与部署"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 训练数据准备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 18:02:39] dataset.py:407 [t:8149425664]: no data source was provided, construct\n",
      "[INFO] [06-12 18:02:39] dataset.py:275 [t:8149425664]: construct a file data source from path: ./1.jsonl, with args: {}\n",
      "[INFO] [06-12 18:02:39] file.py:293 [t:8149425664]: use format type FormatType.Jsonl\n",
      "[INFO] [06-12 18:02:39] utils.py:348 [t:8149425664]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/Desktop/sdk_3.12/bce-qianfan-sdk/cookbook/awesome_demo/role_play/1.arrow\n",
      "[INFO] [06-12 18:02:39] utils.py:276 [t:8149425664]: has got a memory-mapped table\n",
      "[INFO] [06-12 18:02:39] dataset.py:480 [t:8149425664]: no destination data source was provided, construct\n",
      "[INFO] [06-12 18:02:39] dataset.py:275 [t:8149425664]: construct a file data source from path: ./1.jsonl, with args: {}\n",
      "[INFO] [06-12 18:02:39] file.py:293 [t:8149425664]: use format type FormatType.Jsonl\n",
      "[INFO] [06-12 18:02:39] dataset.py:993 [t:8149425664]: list local dataset data by slice(0, 9999, None)\n"
     ]
    }
   ],
   "source": [
    "ds = Dataset.load(data_file='./data/train.jsonl')\n",
    "\n",
    "ds = ds.save(data_file='./data/train.jsonl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.1 训练数据分析"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 0\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 1\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 2\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 3\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 4\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 5\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 6\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 7\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 8\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 9\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 10\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 11\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 12\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 13\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 14\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 15\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 16\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 17\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 18\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 19\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 20\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 21\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 22\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 23\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 24\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 25\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 26\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 27\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 28\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 29\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 30\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 31\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 32\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 33\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 34\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 35\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 36\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 37\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 38\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 39\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 40\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 41\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 42\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 43\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 44\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 45\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 46\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 47\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 48\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 49\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 50\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 51\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 52\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 53\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 54\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 55\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 56\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 57\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 58\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 59\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 60\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 61\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 62\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 63\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 64\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 65\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 66\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 67\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 68\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 69\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 70\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 71\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 72\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 73\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 74\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 75\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 76\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 77\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 78\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 79\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 80\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 81\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 82\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 83\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 84\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 85\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 86\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 87\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 88\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 89\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 90\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 91\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 92\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 93\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 94\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 95\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 96\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 97\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 98\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 99\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 100\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 101\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 102\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 103\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 104\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 105\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 106\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 107\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 108\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 109\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 110\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 111\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 112\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 113\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 114\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 115\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 116\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 117\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 118\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 119\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 120\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 121\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 122\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 123\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 124\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 125\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 126\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 127\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 128\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 129\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 130\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 131\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 132\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 133\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 134\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 135\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 136\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 137\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 138\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 139\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 140\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 141\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 142\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 143\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 144\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 145\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 146\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 147\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 148\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 149\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 150\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 151\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 152\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 153\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 154\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 155\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 156\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 157\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 158\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 159\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 160\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 161\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 162\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 163\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 164\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 165\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 166\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 167\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 168\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 169\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 170\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 171\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 172\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 173\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 174\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 175\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 176\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 177\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 178\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 179\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 180\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 181\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 182\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 183\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 184\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 185\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 186\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 187\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 188\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 189\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 190\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 191\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 192\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 193\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 194\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 195\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 196\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 197\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 198\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 199\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 200\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 201\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 202\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 203\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 204\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 205\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 206\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 207\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 208\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 209\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 210\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 211\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 212\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 213\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 214\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 215\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 216\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 217\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 218\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 219\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 220\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 221\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 222\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 223\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 224\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 225\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 226\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 227\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 228\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 229\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 230\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 231\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 232\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 233\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 234\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 235\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 236\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 237\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 238\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 239\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 240\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 241\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 242\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 243\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 244\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 245\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 246\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 247\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 248\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 249\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 250\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 251\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 252\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 253\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 254\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 255\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 256\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 257\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 258\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 259\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 260\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 261\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 262\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 263\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 264\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 265\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 266\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 267\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 268\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 269\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 270\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 271\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 272\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 273\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 274\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 275\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 276\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 277\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 278\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 279\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 280\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 281\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 282\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 283\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 284\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 285\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 286\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 287\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 288\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 289\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 290\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 291\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 292\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 293\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 294\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 295\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 296\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 297\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 298\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 299\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 300\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 301\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 302\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 303\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 304\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 305\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 306\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 307\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 308\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 309\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 310\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 311\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 312\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 313\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 314\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 315\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 316\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 317\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 318\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 319\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 320\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 321\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 322\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 323\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 324\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 325\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 326\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 327\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 328\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 329\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 330\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 331\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 332\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 333\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 334\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 335\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 336\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 337\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 338\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 339\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 340\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 341\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 342\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 343\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 344\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 345\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 346\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 347\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 348\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 349\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 350\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 351\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 352\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 353\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 354\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 355\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 356\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 357\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 358\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 359\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 360\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 361\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 362\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 363\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 364\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 365\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 366\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 367\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 368\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 369\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 370\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 371\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 372\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 373\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 374\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 375\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 376\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 377\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 378\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 379\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 380\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 381\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 382\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 383\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 384\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 385\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 386\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 387\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 388\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 389\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 390\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 391\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 392\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 393\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 394\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 395\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 396\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 397\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 398\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 399\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 400\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 401\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 402\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 403\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 404\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 405\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 406\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 407\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 408\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 409\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 410\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 411\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 412\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 413\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 414\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 415\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 416\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 417\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 418\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 419\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 420\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 421\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 422\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 423\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 424\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 425\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 426\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 427\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 428\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 429\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 430\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 431\n",
      "[INFO] [06-12 18:02:42] dataset.py:993 [t:8149425664]: list local dataset data by 432\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "defaultdict(<class 'int'>, {'zhubajie': 67, 'jiabaoyu': 86, 'likui': 28, 'tangsanzang': 65, 'songjiang': 35, 'sunwukong': 89, 'wusong': 47, 'wuyong': 15})\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAAJfCAYAAACqgZ1yAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB4eElEQVR4nO3deZxO5f/H8fc9gzG2sS9jHfu+L6Hs2bcSRmRfksgS2VW2tEghImuRECpECNllX0OWUPYwGAYzn98ffu7vuRsKae6Z8Xo+HvOoOec693zmcs+5z/uc61zHZWYmAAAAAIAkycfbBQAAAABAdEJIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCANzT1KlT5XK5tGXLFm+XAgBAlCIkAQBincWLF+vNN9/0dhmRDBs2TAsWLHjk7fft26c333xTx44de2w1AQAiIyQBAGKdxYsX66233vJ2GZE8jpD01ltvEZIA4D9GSAIAeFVoaKi3S3ggZqbr1697uwwAQBQgJAHAE+r3339XmzZtFBgYKD8/PwUFBaljx466efOmR7uwsDB1795dqVKlUsKECfXcc8/p3LlzHm2++eYb1apVy/1a2bJl0+DBgxUeHu7RrkKFCsqfP7+2bt2qcuXKKUGCBOrbt+9DvYYkbdq0STVr1lSyZMmUMGFCFSxYUB999JEkqWXLlho7dqwkyeVyub/uioiI0KhRo5QvXz7Fjx9fadKkUYcOHXTx4kWPn5ElSxbVrl1bS5cuVfHixeXv769PP/30vv156NAhNWjQQGnTplX8+PGVIUMGBQcH6/Lly+5arl27pmnTprlratmypSTpt99+0yuvvKJcuXLJ399fKVKkUMOGDT2uGE2dOlUNGzaUJFWsWNH9GqtWrZIkbdmyRdWqVVPKlCnl7++voKAgtW7d+r71AgDuL463CwAARL0//vhDJUuW1KVLl9S+fXvlzp1bv//+u+bOnavQ0FDFixfP3bZz585KliyZBg0apGPHjmnUqFF69dVX9dVXX7nbTJ06VYkSJVL37t2VKFEi/fjjjxo4cKBCQkL03nvvefzsCxcuqEaNGgoODlazZs2UJk2ah3qNZcuWqXbt2kqXLp1ee+01pU2bVvv379fChQv12muvqUOHDvrjjz+0bNkyff7555F+9w4dOmjq1Klq1aqVunTpoqNHj2rMmDHavn271q1bp7hx47rbHjhwQE2aNFGHDh3Url075cqV6579efPmTVWrVk1hYWHq3Lmz0qZNq99//10LFy7UpUuXFBAQoM8//1xt27ZVyZIl1b59e0lStmzZJEk///yz1q9fr+DgYGXIkEHHjh3TuHHjVKFCBe3bt08JEiRQuXLl1KVLF3388cfq27ev8uTJI0nKkyePzp49q6pVqypVqlTq3bu3kiZNqmPHjmnevHkP9b4AAPw/AwA8cZo3b24+Pj72888/R1oXERFhZmZTpkwxSValShX3MjOzbt26ma+vr126dMm9LDQ0NNLrdOjQwRIkSGA3btxwLytfvrxJsvHjx0dq/yCvcfv2bQsKCrLMmTPbxYsX71m3mVmnTp3sXh9xa9asMUk2Y8YMj+VLliyJtDxz5swmyZYsWRLpdf5q+/btJsnmzJnzt+0SJkxoLVq0iLT8Xr/7hg0bTJJNnz7dvWzOnDkmyVauXOnRdv78+Sbpnv+eAICHx3A7AHjCREREaMGCBapTp46KFy8eab1zaJoktW/f3mPZM888o/DwcP3222/uZf7+/u7/v3Llis6fP69nnnlGoaGh+uWXXzxez8/PT61atYr0cx/kNbZv366jR4+qa9euSpo06d/WfS9z5sxRQECAnn32WZ0/f979VaxYMSVKlEgrV670aB8UFKRq1ar94+sGBARIkpYuXfpI91g5f/dbt27pwoULyp49u5ImTapt27b94/Z3+2LhwoW6devWQ/98AIAnQhIAPGHOnTunkJAQ5c+f/4HaZ8qUyeP7ZMmSSZLHPTx79+7Vc889p4CAACVJkkSpUqVSs2bNJMl9T85d6dOn9xjO9zCvcfjwYUl64Nr/6tChQ7p8+bJSp06tVKlSeXxdvXpVZ8+e9WgfFBT0QK8bFBSk7t2767PPPlPKlClVrVo1jR07NtLvfj/Xr1/XwIEDlTFjRvn5+SllypRKlSqVLl269ECvUb58eTVo0EBvvfWWUqZMqXr16mnKlCkKCwt7oJ8PAPDEPUkAgL/l6+t7z+VmJkm6dOmSypcvryRJkujtt99WtmzZFD9+fG3btk1vvPGGIiIiPLZzXjW562Ff41FFREQoderUmjFjxj3Xp0qV6h9rvZ8PPvhALVu21DfffKMffvhBXbp00fDhw7Vx40ZlyJDhb7ft3LmzpkyZoq5du6p06dIKCAiQy+VScHDwA/3uLpdLc+fO1caNG/Xdd99p6dKlat26tT744ANt3LhRiRIleuDfAwBASAKAJ06qVKmUJEkS7dmz57G83qpVq3ThwgXNmzdP5cqVcy8/evToY3+NuxMd7NmzR1WqVLnv691v6F22bNm0fPlylS1b9qEC0IMqUKCAChQooP79+2v9+vUqW7asxo8fryFDhvxtXXPnzlWLFi30wQcfuJfduHFDly5d8mj3T0MKn3rqKT311FMaOnSoZs6cqaZNm2rWrFlq27btv/vFAOAJw3A7AHjC+Pj4qH79+vruu++0ZcuWSOvvXiF6UHevNDm3u3nzpj755JPH/hpFixZVUFCQRo0aFSlAOLdNmDChJEVq06hRI4WHh2vw4MGRarh9+3ak9g8qJCREt2/f9lhWoEAB+fj4eAx5S5gw4T1/hq+vb6R+Hz16dKTpz+/3e128eDHS9oULF5YkhtwBwCPgShIAPIGGDRumH374QeXLl1f79u2VJ08enTp1SnPmzNHatWsjTYrwd8qUKaNkyZKpRYsW6tKli1wulz7//POHClsP+ho+Pj4aN26c6tSpo8KFC6tVq1ZKly6dfvnlF+3du1dLly6VJBUrVkyS1KVLF1WrVk2+vr4KDg5W+fLl1aFDBw0fPlw7duxQ1apVFTduXB06dEhz5szRRx99pBdeeOGB677rxx9/1KuvvqqGDRsqZ86cun37tj7//HP5+vqqQYMG7nbFihXT8uXLNXLkSAUGBiooKEilSpVS7dq19fnnnysgIEB58+bVhg0btHz5cqVIkcLj5xQuXFi+vr4aMWKELl++LD8/P1WqVEkzZ87UJ598oueee07ZsmXTlStXNHHiRCVJkkQ1a9Z86N8HAJ54XptXDwDgVb/99ps1b97cUqVKZX5+fpY1a1br1KmThYWFmdn/pgD/67TSK1eujDQN9bp16+ypp54yf39/CwwMtF69etnSpUsjtStfvrzly5fvnvU86GuYma1du9aeffZZS5w4sSVMmNAKFixoo0ePdq+/ffu2de7c2VKlSmUulyvSdOATJkywYsWKmb+/vyVOnNgKFChgvXr1sj/++MPdJnPmzFarVq0H6ssjR45Y69atLVu2bBY/fnxLnjy5VaxY0ZYvX+7R7pdffrFy5cqZv7+/SXJPB37x4kVr1aqVpUyZ0hIlSmTVqlWzX375xTJnzhxpyvCJEyda1qxZzdfX190327ZtsyZNmlimTJnMz8/PUqdObbVr17YtW7Y8UP0AAE8us4ccVwEAAAAAsRj3JAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwCHWP0w2IiJCf/zxhxInTiyXy+XtcgAAAAB4iZnpypUrCgwMlI/P/a8XxfqQ9McffyhjxozeLgMAAABANHHixAllyJDhvutjfUhKnDixpDsdkSRJEi9XAwAAAMBbQkJClDFjRndGuJ9YH5LuDrFLkiQJIQkAAADAP96Gw8QNAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAIBYLzw8XAMGDFBQUJD8/f2VLVs2DR48WGbmbnPmzBm1bNlSgYGBSpAggapXr65Dhw55sWoA3hLrn5MEAAAwYsQIjRs3TtOmTVO+fPm0ZcsWtWrVSgEBAerSpYvMTPXr11fcuHH1zTffKEmSJBo5cqSqVKmiffv2KWHChN7+FQBEIUISAACI9davX6969eqpVq1akqQsWbLoyy+/1ObNmyVJhw4d0saNG7Vnzx7ly5dPkjRu3DilTZtWX375pdq2beu12gFEPYbbAQCAWK9MmTJasWKFDh48KEnauXOn1q5dqxo1akiSwsLCJEnx48d3b+Pj4yM/Pz+tXbs26gsG4FVcSQIAALFe7969FRISoty5c8vX11fh4eEaOnSomjZtKknKnTu3MmXKpD59+ujTTz9VwoQJ9eGHH+rkyZM6deqUl6sHENW4kgQAAGK92bNna8aMGZo5c6a2bdumadOm6f3339e0adMkSXHjxtW8efN08OBBJU+eXAkSJNDKlStVo0YN+fhwuAQ8abiSBAAAYr2ePXuqd+/eCg4OliQVKFBAv/32m4YPH64WLVpIkooVK6YdO3bo8uXLunnzplKlSqVSpUqpePHi3iwdgBdwagQAAMR6oaGhka4I+fr6KiIiIlLbgIAApUqVSocOHdKWLVtUr169qCoTQDTBlSQAABDr1alTR0OHDlWmTJmUL18+bd++XSNHjlTr1q3dbebMmaNUqVIpU6ZM2r17t1577TXVr19fVatW9WLlALyBkAQAAGK90aNHa8CAAXrllVd09uxZBQYGqkOHDho4cKC7zalTp9S9e3edOXNG6dKlU/PmzTVgwAAvVg3AW1zmfNR0LBQSEqKAgABdvnxZSZIk8XY5AAAAALzkQbMB9yQBAAAAgAMhCQAAAAAcuCcJAABEuSy9F3m7hGjt2Du1vF0C8ETjShIAAAAAOBCSgMckPDxcAwYMUFBQkPz9/ZUtWzYNHjxYf50bZf/+/apbt64CAgKUMGFClShRQsePH/dS1QAAAPgrhtsBj8mIESM0btw4TZs2Tfny5dOWLVvUqlUrBQQEqEuXLpKkw4cP6+mnn1abNm301ltvKUmSJNq7d6/ix4/v5eoBAABwFyEJeEzWr1+vevXqqVatO+PIs2TJoi+//FKbN292t+nXr59q1qypd999170sW7ZsUV4rAAAA7o/hdsBjUqZMGa1YsUIHDx6UJO3cuVNr165VjRo1JEkRERFatGiRcubMqWrVqil16tQqVaqUFixY4MWqAQAA8FeEJOAx6d27t4KDg5U7d27FjRtXRYoUUdeuXdW0aVNJ0tmzZ3X16lW98847ql69un744Qc999xzev7557V69WovVw8AAIC7GG4HPCazZ8/WjBkzNHPmTOXLl087duxQ165dFRgYqBYtWigiIkKSVK9ePXXr1k2SVLhwYa1fv17jx49X+fLlvVk+AAAA/h8hCXhMevbs6b6aJEkFChTQb7/9puHDh6tFixZKmTKl4sSJo7x583pslydPHq1du9YbJQMAAOAeGG4HPCahoaHy8fH8k/L19XVfQYoXL55KlCihAwcOeLQ5ePCgMmfOHGV1AgAA4O9xJQl4TOrUqaOhQ4cqU6ZMypcvn7Zv366RI0eqdevW7jY9e/ZU48aNVa5cOVWsWFFLlizRd999p1WrVnmvcAAAAHggJAGPyejRozVgwAC98sorOnv2rAIDA9WhQwcNHDjQ3ea5557T+PHjNXz4cHXp0kW5cuXS119/raefftqLlQMAAMDJZWbm7SL+SyEhIQoICNDly5eVJEkSb5cDAAAkZem9yNslRGvH3qnl7RKAWOlBswH3JAEAAACAAyEJAAAAABy4JwmxEsM4/hlDOQAAAO6NK0kAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcvBqSwsPDNWDAAAUFBcnf31/ZsmXT4MGDZWbuNmamgQMHKl26dPL391eVKlV06NAhL1YNAAAAIDbzakgaMWKExo0bpzFjxmj//v0aMWKE3n33XY0ePdrd5t1339XHH3+s8ePHa9OmTUqYMKGqVaumGzdueLFyAAAAALFVHG/+8PXr16tevXqqVauWJClLliz68ssvtXnzZkl3riKNGjVK/fv3V7169SRJ06dPV5o0abRgwQIFBwd7rXYAAAAAsZNXrySVKVNGK1as0MGDByVJO3fu1Nq1a1WjRg1J0tGjR3X69GlVqVLFvU1AQIBKlSqlDRs2eKVmAAAAALGbV68k9e7dWyEhIcqdO7d8fX0VHh6uoUOHqmnTppKk06dPS5LSpEnjsV2aNGnc6/4qLCxMYWFh7u9DQkL+o+oBAAAAxEZevZI0e/ZszZgxQzNnztS2bds0bdo0vf/++5o2bdojv+bw4cMVEBDg/sqYMeNjrBgAAABAbOfVkNSzZ0/17t1bwcHBKlCggF566SV169ZNw4cPlySlTZtWknTmzBmP7c6cOeNe91d9+vTR5cuX3V8nTpz4b38JAAAAALGKV0NSaGiofHw8S/D19VVERIQkKSgoSGnTptWKFSvc60NCQrRp0yaVLl36nq/p5+enJEmSeHwBAAAAwIPy6j1JderU0dChQ5UpUybly5dP27dv18iRI9W6dWtJksvlUteuXTVkyBDlyJFDQUFBGjBggAIDA1W/fn1vlg4AAAAglvJqSBo9erQGDBigV155RWfPnlVgYKA6dOiggQMHutv06tVL165dU/v27XXp0iU9/fTTWrJkieLHj+/FygEAAADEVi4zM28X8V8KCQlRQECALl++zNC7J0iW3ou8XUK0d+ydWt4uAcATjP3032MfDfw3HjQbePWeJAAAAACIbghJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQ9AT4/fff1axZM6VIkUL+/v4qUKCAtmzZ4l7vcrnu+fXee+95sWoAAADAO+J4uwD8ty5evKiyZcuqYsWK+v7775UqVSodOnRIyZIlc7c5deqUxzbff/+92rRpowYNGkR1uQAAAIDXEZJiuREjRihjxoyaMmWKe1lQUJBHm7Rp03p8/80336hixYrKmjVrlNQIAAAARCcMt4vlvv32WxUvXlwNGzZU6tSpVaRIEU2cOPG+7c+cOaNFixapTZs2UVglAAAAEH0QkmK5I0eOaNy4ccqRI4eWLl2qjh07qkuXLpo2bdo920+bNk2JEyfW888/H8WVAgAAANEDw+1iuYiICBUvXlzDhg2TJBUpUkR79uzR+PHj1aJFi0jtJ0+erKZNmyp+/PhRXSoAAAAQLXAlKZZLly6d8ubN67EsT548On78eKS2a9as0YEDB9S2bduoKg8AAACIdghJsVzZsmV14MABj2UHDx5U5syZI7WdNGmSihUrpkKFCkVVeQAAAEC0Q0iK5bp166aNGzdq2LBh+vXXXzVz5kxNmDBBnTp18mgXEhKiOXPmcBUJAAAATzxCUixXokQJzZ8/X19++aXy58+vwYMHa9SoUWratKlHu1mzZsnM1KRJEy9VCgAAAEQPhKQnQO3atbV7927duHFD+/fvV7t27SK1ad++vUJDQxUQEOCFCgFER7///ruaNWumFClSyN/fXwUKFNCWLVvc61u2bCmXy+XxVb16dS9WDADA48HsdgCASC5evKiyZcuqYsWK+v7775UqVSodOnRIyZIl82hXvXp1j4dV+/n5RXWpAAA8doQkAEAkI0aMUMaMGT0CUFBQUKR2fn5+Sps2bVSWBgDAf46QFMWy9F7k7RKitWPv1PJ2CQAkffvtt6pWrZoaNmyo1atXK3369HrllVciDdddtWqVUqdOrWTJkqlSpUoaMmSIUqRI4aWqAQB4PLgnCQAQyZEjRzRu3DjlyJFDS5cuVceOHdWlSxdNmzbN3aZ69eqaPn26VqxYoREjRmj16tWqUaOGwsPDvVg5AAD/HleSAACRREREqHjx4ho2bJgkqUiRItqzZ4/Gjx+vFi1aSJKCg4Pd7QsUKKCCBQsqW7ZsWrVqlSpXruyVugEAeBy4kgQAiCRdunTKmzevx7I8efLo+PHj990ma9asSpkypX799df/ujwAAP5ThCQAQCRly5bVgQMHPJYdPHhQmTNnvu82J0+e1IULF5QuXbr/ujwAAP5ThCQAQCTdunXTxo0bNWzYMP3666+aOXOmJkyYoE6dOkmSrl69qp49e2rjxo06duyYVqxYoXr16il79uyqVq2al6sHAODfISQBACIpUaKE5s+fry+//FL58+fX4MGDNWrUKDVt2lSS5Ovrq127dqlu3brKmTOn2rRpo2LFimnNmjU8KwkAEOMxcQMA4J5q166t2rVr33Odv7+/li5dGsUVAQAQNbiSBAAAAAAOhCQAAAAAcGC4HQDEAFl6L/J2CdHasXdqebsEAEAswpUkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcvB6Sfv/9dzVr1kwpUqSQv7+/ChQooC1btrjXm5kGDhyodOnSyd/fX1WqVNGhQ4e8WDEAAACA2MyrIenixYsqW7as4saNq++//1779u3TBx98oGTJkrnbvPvuu/r44481fvx4bdq0SQkTJlS1atV048YNL1YOAAAAILaK480fPmLECGXMmFFTpkxxLwsKCnL/v5lp1KhR6t+/v+rVqydJmj59utKkSaMFCxYoODg4ymsGAAAAELt59UrSt99+q+LFi6thw4ZKnTq1ihQpookTJ7rXHz16VKdPn1aVKlXcywICAlSqVClt2LDBGyUDAAAAiOW8GpKOHDmicePGKUeOHFq6dKk6duyoLl26aNq0aZKk06dPS5LSpEnjsV2aNGnc6/4qLCxMISEhHl8AAAAA8KC8OtwuIiJCxYsX17BhwyRJRYoU0Z49ezR+/Hi1aNHikV5z+PDheuuttx5nmQAAAACeIF69kpQuXTrlzZvXY1mePHl0/PhxSVLatGklSWfOnPFoc+bMGfe6v+rTp48uX77s/jpx4sR/UDkAAACA2MqrIals2bI6cOCAx7KDBw8qc+bMku5M4pA2bVqtWLHCvT4kJESbNm1S6dKl7/mafn5+SpIkiccXAAAAADworw6369atm8qUKaNhw4apUaNG2rx5syZMmKAJEyZIklwul7p27aohQ4YoR44cCgoK0oABAxQYGKj69et7s3QAAAAAsZRXQ1KJEiU0f/589enTR2+//baCgoI0atQoNW3a1N2mV69eunbtmtq3b69Lly7p6aef1pIlSxQ/fnwvVg4AAAAgtvJqSJKk2rVrq3bt2vdd73K59Pbbb+vtt9+OwqoAAAAAPKm8ek8SAAAAAEQ3hCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHB4pJBUqVIlXbp0KdLykJAQVapU6d/WBAAAAABe80ghadWqVbp582ak5Tdu3NCaNWv+dVEAAAAA4C1xHqbxrl273P+/b98+nT592v19eHi4lixZovTp0z++6gAAAAAgij1USCpcuLBcLpdcLtc9h9X5+/tr9OjRj604AAAAxFzvvPOO+vTpo9dee02jRo3SsWPHFBQUdM+2s2fPVsOGDaO4QuDeHiokHT16VGamrFmzavPmzUqVKpV7Xbx48ZQ6dWr5+vo+9iIBAAAQs/z888/69NNPVbBgQfeyjBkz6tSpUx7tJkyYoPfee081atSI6hKB+3qokJQ5c2ZJUkRExH9SDAAAAGK+q1evqmnTppo4caKGDBniXu7r66u0adN6tJ0/f74aNWqkRIkSRXWZwH09VEhyOnTokFauXKmzZ89GCk0DBw7814UBAAAgZurUqZNq1aqlKlWqeISkv9q6dat27NihsWPHRmF1wD97pJA0ceJEdezYUSlTplTatGnlcrnc61wuFyEJAADgCTVr1ixt27ZNP//88z+2nTRpkvLkyaMyZcpEQWXAg3ukkDRkyBANHTpUb7zxxuOuBwAAADHUiRMn9Nprr2nZsmWKHz/+37a9fv26Zs6cqQEDBkRRdcCDe6SQdPHiRWYfAQAAgIetW7fq7NmzKlq0qHtZeHi4fvrpJ40ZM0ZhYWHuSb7mzp2r0NBQNW/e3FvlAvf1SA+TbdiwoX744YfHXQsAAABisMqVK2v37t3asWOH+6t48eJq2rSpduzY4TEL8qRJk1S3bl2P2ZKB6OKRriRlz55dAwYM0MaNG1WgQAHFjRvXY32XLl0eS3EAAACIORInTqz8+fN7LEuYMKFSpEjhsfzXX3/VTz/9pMWLF0d1icADeaSQNGHCBCVKlEirV6/W6tWrPda5XC5CEgAAAO5r8uTJypAhg6pWrertUoB7eqSQdPTo0cddBwAAAGKhVatWRVo2bNgwDRs2LOqLAR7QI92TBAAAAACx1SNdSWrduvXfrp88efIjFQMAAAAA3vbIU4A73bp1S3v27NGlS5dUqVKlx1IYAAAA/p0svRd5u4Ro7dg7tbxdAqKpRwpJ8+fPj7QsIiJCHTt2VLZs2f51UQAAAADgLY/tniQfHx91795dH3744eN6SQAAAACIco914obDhw/r9u3bj/MlAQAAACBKPdJwu+7du3t8b2Y6deqUFi1apBYtWjyWwgAAAADAGx4pJG3fvt3jex8fH6VKlUoffPDBP858BwAAAADR2SOFpJUrVz7uOgAAAAAgWnikkHTXuXPndODAAUlSrly5lCpVqsdSFAAAAAB4yyNN3HDt2jW1bt1a6dKlU7ly5VSuXDkFBgaqTZs2Cg0Nfdw1AgAAAECUeaSQ1L17d61evVrfffedLl26pEuXLumbb77R6tWr1aNHj8ddIwAAAABEmUcabvf1119r7ty5qlChgntZzZo15e/vr0aNGmncuHGPqz4AAAAAiFKPdCUpNDRUadKkibQ8derUDLcDAAAAEKM9UkgqXbq0Bg0apBs3briXXb9+XW+99ZZKly792IoDAAAAgKj2SMPtRo0aperVqytDhgwqVKiQJGnnzp3y8/PTDz/88FgLBAAAAICo9EghqUCBAjp06JBmzJihX375RZLUpEkTNW3aVP7+/o+1QAAAAACISo8UkoYPH640adKoXbt2HssnT56sc+fO6Y033ngsxQEAAABAVHuke5I+/fRT5c6dO9LyfPnyafz48f+6KAAAAADwlkcKSadPn1a6dOkiLU+VKpVOnTr1r4sCAAAAAG95pJCUMWNGrVu3LtLydevWKTAw8F8XBQAAAADe8kj3JLVr105du3bVrVu3VKlSJUnSihUr1KtXL/Xo0eOxFggAAAAAUemRQlLPnj114cIFvfLKK7p586YkKX78+HrjjTfUp0+fx1ogAAAAAESlRwpJLpdLI0aM0IABA7R//375+/srR44c8vPze9z1AQAAAECUeqSQdFeiRIlUokSJx1ULAAAAAHjdI03cAAAAAACxFSEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAO0SYkvfPOO3K5XOratat72Y0bN9SpUyelSJFCiRIlUoMGDXTmzBnvFQkAAAAg1osWIennn3/Wp59+qoIFC3os79atm7777jvNmTNHq1ev1h9//KHnn3/eS1UCAAAAeBJ4PSRdvXpVTZs21cSJE5UsWTL38suXL2vSpEkaOXKkKlWqpGLFimnKlClav369Nm7c6MWKAQAAAMRmXg9JnTp1Uq1atVSlShWP5Vu3btWtW7c8lufOnVuZMmXShg0borpMAAAAAE+ION784bNmzdK2bdv0888/R1p3+vRpxYsXT0mTJvVYniZNGp0+ffq+rxkWFqawsDD39yEhIY+tXgAAAACxn9euJJ04cUKvvfaaZsyYofjx4z+21x0+fLgCAgLcXxkzZnxsrw0AAAAg9vNaSNq6davOnj2rokWLKk6cOIoTJ45Wr16tjz/+WHHixFGaNGl08+ZNXbp0yWO7M2fOKG3atPd93T59+ujy5cvurxMnTvzHvwkAAACA2MRrw+0qV66s3bt3eyxr1aqVcufOrTfeeEMZM2ZU3LhxtWLFCjVo0ECSdODAAR0/flylS5e+7+v6+fnJz8/vP60dAAAAQOzltZCUOHFi5c+f32NZwoQJlSJFCvfyNm3aqHv37kqePLmSJEmizp07q3Tp0nrqqae8UTIAAACAJ4BXJ274Jx9++KF8fHzUoEEDhYWFqVq1avrkk0+8XRYAAACAWCxahaRVq1Z5fB8/fnyNHTtWY8eO9U5BAAAAAJ44Xn9OEgAAAABEJ4QkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAMCLxo0bp4IFCypJkiRKkiSJSpcure+//969vkKFCnK5XB5fL7/8shcrBoDYL1pNAQ4AwJMmQ4YMeuedd5QjRw6ZmaZNm6Z69epp+/btypcvnySpXbt2evvtt93bJEiQwFvlAsATgZAEAIAX1alTx+P7oUOHaty4cdq4caM7JCVIkEBp06b1RnkA8ERiuB0AANFEeHi4Zs2apWvXrql06dLu5TNmzFDKlCmVP39+9enTR6GhoV6sEgBiP64kAQDgZbt371bp0qV148YNJUqUSPPnz1fevHklSS+++KIyZ86swMBA7dq1S2+88YYOHDigefPmeblqAIi9CEkAAHhZrly5tGPHDl2+fFlz585VixYttHr1auXNm1ft27d3tytQoIDSpUunypUr6/Dhw8qWLZsXqwaA2IvhdgAAeFm8ePGUPXt2FStWTMOHD1ehQoX00Ucf3bNtqVKlJEm//vprVJYIAE8UQhIAANFMRESEwsLC7rlux44dkqR06dJFYUUA8GRhuB0AAF7Up08f1ahRQ5kyZdKVK1c0c+ZMrVq1SkuXLtXhw4c1c+ZM1axZUylSpNCuXbvUrVs3lStXTgULFvR26QAQaxGSAADworNnz6p58+Y6deqUAgICVLBgQS1dulTPPvusTpw4oeXLl2vUqFG6du2aMmbMqAYNGqh///7eLhsAYjVCEgAAXjRp0qT7rsuYMaNWr14dhdUAACTuSQIAAAAAD4QkAAAAAHBguB0AAP8vS+9F3i4h2jv2Ti1vlwAA/zmuJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAMQow4cPV4kSJZQ4cWKlTp1a9evX14EDBzzanD59Wi+99JLSpk2rhAkTqmjRovr666+9VDEAAIhpCEkAYpTVq1erU6dO2rhxo5YtW6Zbt26patWqunbtmrtN8+bNdeDAAX377bfavXu3nn/+eTVq1Ejbt2/3YuUAACCmiOPtAgDgYSxZssTj+6lTpyp16tTaunWrypUrJ0lav369xo0bp5IlS0qS+vfvrw8//FBbt25VkSJForxmAAAQs3AlCUCMdvnyZUlS8uTJ3cvKlCmjr776Sn/++aciIiI0a9Ys3bhxQxUqVPBSlQAAICbhShKAGCsiIkJdu3ZV2bJllT9/fvfy2bNnq3HjxkqRIoXixImjBAkSaP78+cqePbsXqwUAADEFIQlAjNWpUyft2bNHa9eu9Vg+YMAAXbp0ScuXL1fKlCm1YMECNWrUSGvWrFGBAgW8VC0AAIgpCEkAYqRXX31VCxcu1E8//aQMGTK4lx8+fFhjxozRnj17lC9fPklSoUKFtGbNGo0dO1bjx4/3VskAACCGICQBiFHMTJ07d9b8+fO1atUqBQUFeawPDQ2VJPn4eN5y6evrq4iIiCirEwAAxFyEJAAxSqdOnTRz5kx98803Spw4sU6fPi1JCggIkL+/v3Lnzq3s2bOrQ4cOev/995UiRQotWLBAy5Yt08KFC71cPQAAiAmY3Q5AjDJu3DhdvnxZFSpUULp06dxfX331lSQpbty4Wrx4sVKlSqU6deqoYMGCmj59uqZNm6aaNWt6uXoAABATcCUJQIxiZv/YJkeOHPr666+joBoAABAbcSUJAAAAABwISQAAAADgwHA7AP9Klt6LvF1CtHbsnVreLgEAADwkriQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABy8GpKGDx+uEiVKKHHixEqdOrXq16+vAwcOeLS5ceOGOnXqpBQpUihRokRq0KCBzpw546WKAQAAAMR2Xg1Jq1evVqdOnbRx40YtW7ZMt27dUtWqVXXt2jV3m27duum7777TnDlztHr1av3xxx96/vnnvVg1AAAAgNgsjjd/+JIlSzy+nzp1qlKnTq2tW7eqXLlyunz5siZNmqSZM2eqUqVKkqQpU6YoT5482rhxo5566ilvlA0AAAAgFotW9yRdvnxZkpQ8eXJJ0tatW3Xr1i1VqVLF3SZ37tzKlCmTNmzYcM/XCAsLU0hIiMcXAAAAADyoaBOSIiIi1LVrV5UtW1b58+eXJJ0+fVrx4sVT0qRJPdqmSZNGp0+fvufrDB8+XAEBAe6vjBkz/telAwAAAIhFok1I6tSpk/bs2aNZs2b9q9fp06ePLl++7P46ceLEY6oQAAAAwJPAq/ck3fXqq69q4cKF+umnn5QhQwb38rRp0+rmzZu6dOmSx9WkM2fOKG3atPd8LT8/P/n5+f3XJQMAAACIpbx6JcnM9Oqrr2r+/Pn68ccfFRQU5LG+WLFiihs3rlasWOFeduDAAR0/flylS5eO6nIBAAAAPAG8eiWpU6dOmjlzpr755hslTpzYfZ9RQECA/P39FRAQoDZt2qh79+5Knjy5kiRJos6dO6t06dLMbAcAAADgP+HVkDRu3DhJUoUKFTyWT5kyRS1btpQkffjhh/Lx8VGDBg0UFhamatWq6ZNPPoniSgEAAAA8KbwakszsH9vEjx9fY8eO1dixY6OgIgAAAABPumgzux0AAAAARAeEJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAGKYn376SXXq1FFgYKBcLpcWLFjgsb5ly5ZyuVweX9WrV/dOsTEQIQkAAACIYa5du6ZChQr97WNyqlevrlOnTrm/vvzyyyisMGbz6nOSAAAAADy8GjVqqEaNGn/bxs/PT2nTpo2iimIXriQBAAAAsdCqVauUOnVq5cqVSx07dtSFCxe8XVKMwZUkAAAAIJapXr26nn/+eQUFBenw4cPq27evatSooQ0bNsjX19fb5UV7hCQAAAAglgkODnb/f4ECBVSwYEFly5ZNq1atUuXKlb1YWczAcDsAAAAglsuaNatSpkypX3/91dulxAiEJAAAACCWO3nypC5cuKB06dJ5u5QYgeF2AAAAQAxz9epVj6tCR48e1Y4dO5Q8eXIlT55cb731lho0aKC0adPq8OHD6tWrl7Jnz65q1ap5seqYg5AEAAAAxDBbtmxRxYoV3d93795dktSiRQuNGzdOu3bt0rRp03Tp0iUFBgaqatWqGjx4sPz8/LxVcoxCSAIAAABimAoVKsjM7rt+6dKlUVhN7MM9SQAAAADgQEgCAAAAAAeG2wEAAAD/Qpbei7xdQrR37J1a3i7hoXAlCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAgZAEAAAAAA6EJAAAAABwICQBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAAdCEgAAAAA4EJIAAAAAwIGQBAAAAAAOhCQAAAAAcCAkAQAAAIADIQkAAAAAHAhJAAAAAOBASAIAAAAAB0ISAAAAADgQkgAAAADAIUaEpLFjxypLliyKHz++SpUqpc2bN3u7JAAAAACxVLQPSV999ZW6d++uQYMGadu2bSpUqJCqVaums2fPers0AAAAALFQtA9JI0eOVLt27dSqVSvlzZtX48ePV4IECTR58mRvlwYAAAAgForj7QL+zs2bN7V161b16dPHvczHx0dVqlTRhg0b7rlNWFiYwsLC3N9fvnxZkhQSEvLfFvuAIsJCvV1CtPa4/p3o539GX0cN+jlq0M9Rh76OGvRz1KCfo050ORa/W4eZ/W07l/1TCy/6448/lD59eq1fv16lS5d2L+/Vq5dWr16tTZs2RdrmzTff1FtvvRWVZQIAAACIQU6cOKEMGTLcd320vpL0KPr06aPu3bu7v4+IiNCff/6pFClSyOVyebGy6CckJEQZM2bUiRMnlCRJEm+XE6vR11GDfo4a9HPUoa+jBv0cNejnqENf35+Z6cqVKwoMDPzbdtE6JKVMmVK+vr46c+aMx/IzZ84obdq099zGz89Pfn5+HsuSJk36X5UYKyRJkoQ/oChCX0cN+jlq0M9Rh76OGvRz1KCfow59fW8BAQH/2CZaT9wQL148FStWTCtWrHAvi4iI0IoVKzyG3wEAAADA4xKtryRJUvfu3dWiRQsVL15cJUuW1KhRo3Tt2jW1atXK26UBAAAAiIWifUhq3Lixzp07p4EDB+r06dMqXLiwlixZojRp0ni7tBjPz89PgwYNijQ8EY8ffR016OeoQT9HHfo6atDPUYN+jjr09b8XrWe3AwAAAICoFq3vSQIAAACAqEZIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQgVmNuGgAA8LAISUA0FhER4e0SYrSIiAi5XC5vlwHgCcU+HIi5CEl4aPfb6fNh8Pj5+Nz5E50xY4YOHz7s5Wpilg8++EBt27aVxNWkmOqv/278Oz4a9s3ec3cf/uuvv3q5ktiD/cB/b//+/d4uIVogJOGhREREuHf6ixYt0rhx4/TZZ5/p119/lY+PDx/G/4ErV66oQ4cOatmypY4dO+btcmKE27dvK0WKFHrjjTckyX01ifdnzHL3323Tpk0KDQ2Vy+XiAOkR3N1n//jjj16u5Mnh3NesWLFCOXPm1JIlS7xYUcz31VdfSRKjA/5ja9asUb58+TR//nxvl+J1hCQ8lLsftr169VKXLl00c+ZMffvtt8qXL582bdrkXo9H99eDwMSJE2vfvn36/fff1apVKx09etRLlcUcceLEUcuWLZUrVy6tW7dOwcHBunXrFkE+hnD+Gy1evFjt2rXTuHHjFBYWRlB6CM5+PHDggKpUqaKhQ4d6saIng/Nk4vjx47Vv3z5JUps2bbR48WJvlhZjHT9+XE2aNFGzZs3cy9gPPH5jxozRtm3bJEnt27fXnDlzvFyRd3FEiwfi3Bl9/vnnmj59umbNmqU1a9bo+eef161bt3TkyBEvVhh73D1LdvPmTUl3+j5TpkxatWqVjhw5otatWxOU/sbd92p4eLgiIiK0e/du7dixQ23atNHt27cJStGcmbkPMKdPn67Vq1fr+PHj+uijj/Tpp5/qxo0bBKUH4OzH9957TxMmTFDixIk1YMAADRgwwMvVxW53+71///4aNGiQkiZNqnfffVfFihVT48aNtWjRIi9XGLMMGTJEb775pjJnzqyZM2eqQYMGksR+4DEbMGCA3n77baVOnVofffSRKleurJdeekmzZ8/2dmneY8DfmDdvXqRlAwYMsO7du5uZ2ddff22JEiWyCRMmmJlZSEiInTlzJkprjI0++OADq1mzpoWEhJiZWUREhJmZHTt2zNKnT281a9a0X3/91ZslRnvnzp0zM7MrV67YhAkTrFixYtakSRO7deuWmZmFh4d7szz8gwEDBliyZMlsypQpNnPmTKtYsaIVLFjQRo4cadevXzez//1d4P4GDRpkKVOmtG+//dbmzp1rvXv3Nl9fX+vTp4+3S4vVTp06Zbly5bIpU6a4l127ds1atWpliRMntsWLF3uvuBhk+PDhljRpUlu+fLmtXbvWxo8fbylSpLC6deu627Af+PfOnj1r+fPndx/LmZnduHHDOnfubPHjx7e5c+d6sTrvISThviZMmGClS5e28PBwjwPK119/3Tp37mwLFiywRIkS2bhx48zszo5q8uTJNmjQIPdBDB7NqlWrLEGCBNa0aVN3ULr7b/Dpp5+ay+WyUqVK2R9//OHNMqOtffv2mcvlsvnz55uZ2dWrV+3TTz+1YsWKWXBwsN28edPMCErRUUREhJ08edJy585tU6dOdS+/fv26vfTSS5Y1a1b7+OOPCUoPICQkxMqXL28jR450L7t69aqNHz/eXC6XvfXWW16sLnY7duyYJU2a1BYuXGhm/9vXnD9/3goWLGiBgYG2dOlSj3XwFBYWZvXr17d+/fq5l928edOWLFliAQEB1rhxY/dy9gP/zsmTJy158uQ2e/ZsM7vznoyIiLBLly5ZyZIlLUWKFO7P0yfp/cpwO9xXvXr1tGbNGvn4+Gj79u3u5bly5dKyZcvUrFkzDR8+XC+//LIk6fLly5ozZ45u3bql+PHje6vsGG3w4MH6+OOPVb58eS1btkwLFy5U+/btFRIS4h7CkTBhQrVv314ZMmRQ6tSpvVxx9JQsWTK1aNFCwcHBWrRokRImTKimTZuqffv2OnTokFq3bq2bN29yD1005HK5lChRIvn4+Cg0NFTSnYk44sePr+nTpyt+/PgaM2aMPv30U/c9Sri/w4cP68KFC+7vEyZMqCZNmqhu3bp68803NXjwYC9WFzvca/hu5syZVaxYMY0ZM0ahoaHy8fGRmSkgIEC5cuVSkiRJ1KBBA/ekR8awsUh8fX31+++/e8wMGDduXI9hYMHBwZIYevdvpU+fXk8//bTGjh2rixcvut+TSZIkUc6cOZU+fXoFBwdr586dT9Tn5pPzm+KBtW/fXrdv31bq1Knl6+ur1atXq0SJEhozZowkqW3btsqTJ498fX2VLl06HTt2TPv371eTJk109uxZvfXWW17+DWKuq1evatq0aTp9+rTKlCmjxYsXa+nSpWrbtq127Nihc+fOac6cOSpYsKDmzp0rX19fhYeHe7tsr3N+OJqZ0qZNq3feeUetW7dW3bp1PYLSyy+/rA0bNuiVV17xYsW4614HmPHixVPSpEm1dOlSSXcm4rj7Pi9cuLASJEigOXPmaPXq1ZK4gVu6dz8mTpxYL730klasWKHNmze7lydJkkT58+dXzZo1NWzYMH3yySdRWWqs4pyk4fDhwx735rZv315//vmnXn/9dZmZ+0A+IiJCEyZMUIkSJdSzZ0/dunXLW+VHa76+vmrdurX279+vb7/91r08Tpw4ypUrl1588UWtWbNGr732miRmvXtYBw8e1C+//OL+vk2bNrp586Zef/11Xb9+XT4+Prp9+7auXbvmvkepd+/eCgsLe2L2uYQkeDh48KBOnTrl8QcQFBSkPn36aODAgRo9erQkad68eSpZsqTefPNN5cqVS61bt1ZoaKg2bNjgcUCDh1O1alWFhobq0KFDkqQyZcroxx9/1Lp161SjRg0VLVpUv/32m9q1a+fextfX11vlRhsul0srV67U5s2b3QciadKk0cCBA9WhQwfVrVtXS5YsUcKECRUcHKz+/furX79+3i77iec8wNy1a5eOHz+uU6dOyd/fX6NHj9by5cvVsWNH3bp1y/3vamYaNmyYwsLCNH78eEkcHDn7cceOHVq3bp17H1y9enXFixdPo0ePdgelK1euaM+ePWrQoIE6dOigr776Sn/++ecTc+DzON3t9969e6t27doqUKCA2rdvr507d6phw4Zq2rSpNmzYoLx58+qVV17RU089pUOHDqlMmTLKnj27bt68qbhx4z7x7+G71qxZo++++06nTp2SJFWqVEkZM2bUxIkTNXfuXEnSxYsXtWzZMhUrVkyvvfaaVq5cqZMnT3qz7BjnjTfeUO3atVWkSBE1a9ZMmzdvVt26ddWyZUvt3r1b+fLlU+vWrVWyZEkdPnxY5cqVU7Zs2XT79m35+fk9Oe9XLwzxQwwxefJkCwsLM7M741UHDBhgiRMnto8++sjdZvv27bZw4ULbuXOne5zq3RvjcX/O8dN374+5q06dOvbMM894LDt//rzNnTvX5s6da7dv3zYz+tnsf/149epVe+GFF8zf39+2bNnise7EiRNWvnx5ix8/vvv+AMavRy+9e/e2DBkyWMaMGa1GjRq2evVqMzP75ptvLGHChPbUU09ZvXr1rFSpUpYjRw4zM3vzzTetTJky7r8HmPXs2dNSp05tyZIls5w5c9ry5cvNzOzbb7+1SpUqWYYMGaxcuXKWL18+K1CggJmZvfvuu1aoUCHuI31IzvsyZs2aZVmyZLE5c+bY5MmTLWfOnFa7dm3buHGjmZlt2rTJOnToYC+88IJ17NjRbty4YWZmTZs2tTZt2tjNmzfZJ9md929AQIAFBgZawoQJbdKkSWZ25zijQYMGliFDBsuWLZvlypXL8uXLZ2Z3+j579uzuiXrwz+bMmWPZsmWz+fPn2+zZsy1//vxWtWpVW7VqlZmZ7dixw3r06GEvvviivfbaa+7jwObNm1uLFi0sLCzsiXm/EpJwT6dOnbLkyZNb8eLF3QfxJ06ccAel0aNH33O7J+mGvsdh7NixNmjQINu7d6972dq1a61IkSL2448/mlnkEGVmHBg6LFu2zDZv3my7du2y4OBgS506tW3evNmjzauvvmqJEye2FClS2JUrV56YHXx05ez/lStXWsaMGW3FihU2fvx4Cw4OtqxZs9rKlSvNzOzIkSPWuXNna9OmjXXt2tX999CwYUNr2rTpE/234NzfLl682PLmzWtLliyxvXv3Wo0aNSxTpkzum60PHjxoM2fOtI4dO9o777zjPvBp3769NWjQwK5du+aNXyHGW7VqlfXq1ctjVrCtW7da8eLFrVatWvbTTz9F2ubatWvWo0cPS548ue3bty8qy4221qxZY0WLFrWffvrJzp49az169LCkSZO6T8qeO3fONm7caG+++aZNnjzZvR/o1KmTVa1a1S5fvuzN8mOM5cuXW69eveyTTz5xL9u3b5+VKVPGnn32WVu2bFmkbUJCQqx79+5P5PuVkIT72rJli+XOndueeuqpSEEpadKk9t5773m5wpivTZs2VrlyZUuQIIH169fPlixZYmZmJUuWtK5du3q5uujnwIED7oO78PBwu3jxouXOndsdKPfs2WONGjWyNGnS2NatW93b9ejRw7744gs7f/68V+rGvU2fPt3eeOMNe/fdd93Ltm3bZk2bNrUsWbK4P7CdQej06dPWo0cPS5kypcfJhSfZ1KlTbdiwYTZs2DCP5fXr13cHpbtXL+767bff7PXXX7eAgADbvXt3VJYbK0RERNjhw4ctUaJE5nK57M033/RYv3XrVitRooTVr1/fvvvuO/fyw4cPW8+ePa1gwYK2bdu2qC47Wvr444+tX79+1qtXL4/lb7zxhiVNmtQ+/vhju3jxose6vXv3Wvfu3S1JkiS2c+fOKKw2ZoqIiLATJ05Y0qRJzeVyWe/evT3W79+/38qUKWM1a9b0mO77t99+s969e1uRIkVs+/btUVy19xGScN+rP7dv37YtW7ZY9uzZPYLSyZMn7bXXXrMqVapwRv4h3K+fb968aVOmTLHatWtbxowZrW3btvbqq69aQECAx4H+k27+/Pnmcrls7ty57vdiaGioZcuWzeNs7d69e61x48YWP358e/XVV61x48aWOnVqnisVzRw5csQqVapkCRMmjHSAuW3bNmvWrJlly5bN43kyJ0+etHfeecdy5cr1RH5g30+OHDnM5XJZy5YtI6177rnnLFu2bPbFF1+4TzBcvXrV+vfvb2XKlLEdO3ZEdbkx1r0+71auXGlBQUH27LPPRtpfb9u2zTJlyhTp4H/Xrl126tSp/7TWmOS5554zl8tl1atXt9DQUI91vXv3tpQpU9rw4cM9Hofx4YcfWtWqVQlID2nz5s2WK1cuK1eunHs46F2//PKLZc+e3f0czLv2799vp0+fjsoyow1C0hPOeeA+e/ZsGz58uI0YMcJ9QHm/oHTmzBn3BwZB6Z/9tZ9HjRplw4cPt3379rn79PTp07Zz506rUaOGlSxZ0lwul3344YeRtn+SNWrUyFKkSGHz5s2z69evW2hoqOXJk8cOHjzo0e7333+3t99+28qWLWt169blQDCa+v777+3ZZ5+1wMDASKFn+/btVrNmTWvQoIF7WUREhP3+++88sPr/OfcLlStXtrRp09rq1asjDUF85pln7IUXXvBYdvHiRe7jeAjOvr57/9bdfl6yZIllzpzZWrRoEWlfc+DAAXc79uP317FjR4sXL57NmjUr0lXPjh07WvXq1T2ONe6OJMA/u3btmkVERLiPNdasWWNZs2a1xo0b288//+zR9tixY7xfHQhJTzDnDqdXr16WOXNmK1++vFWvXt1SpUrlHgoQHh7uEZTuThhw92FjeHA9evSwFClSWJUqVSxFihRWuHBhGzx4sMeHQnh4uPs+jNSpUz/xQ8RmzJjhceDRpEkTCwgIsDlz5tiRI0csf/78dvLkyXtue/PmzUgfuIh6f/dh+8MPP1jNmjWtZMmSkQ4wDx486N6WfU3kfvxrGCpWrJjlzJnT1q1bF6mtsx/py4fj7MuRI0fa888/b1WrVrVu3brZ2bNnzezOPWGZM2e25s2b3/PqxpN879y9nDp1yk6cOGFXrlxxL2vSpIklSZLE5s6d677qeZfzpCzv3wf3/vvvW926da18+fLWqVMnO3bsmJmZrV692h2U7k525MT79Q5CEmzs2LGWPn169xmF6dOnm8vlssSJE7uHMYWHh9vPP/9sOXLksPLly3ux2phr/vz5FhgY6B6ScevWLXvttdfsmWeesVGjRllERITHjunMmTNWuHBhmzdvnrdK9rrDhw9b7ty53Tv2u5o0aWKpUqWyCRMmWJYsWaxhw4Y2aNAgGzFihPXr1899DxIfpt7nPMCcOnWqdezY0V5//XX78ssv3csXL15stWrVspIlS97zAJMzmp59MHr0aGvTpo1VqFDBFi5c6DF0q1ixYpYrVy5bv379fYMSHk3v3r0tRYoU9vbbb1vz5s2tZMmSlilTJvv999/N7M6V0axZs1qdOnXs0KFDXq42+howYICVKlXKEidObLVr1/a4l+7uSbB58+ZFOsHF/vzh9OnTx1KkSGEjRoywdu3a2TPPPGNp0qRxj7z46aefLEeOHPbss8/a/v37vVxt9ERIesJdvHjRunTpYtOmTTMzs++++84SJ05s7733njVo0MCSJk3qniksPDzcNmzYYGnSpPGYGQUPZvTo0VaoUCG7evWq+2Dl8uXL9tJLL1mZMmXuuU3OnDlt3LhxUVlmtLFw4UL3WVqzO9OSOsf8N27c2FwulxUqVMiqVatmzZo1s+DgYHv22Wetdu3atmvXLm+Ujfvo1auXBQYGWvPmza158+aWIUMGe//9993rv//+e6tbt65lyZKF+8cc/hpsevfubWnSpLFu3bpZx44dLVmyZPb222979FnJkiWZkOEx279/v+XMmdO+//5797K9e/dapUqVLHfu3O6hX9999509//zzBNL7ePvtty158uQ2e/Zsmz59unXu3NmyZMnicR/MSy+9ZC6Xyz0lNR7eoUOHLHfu3LZo0SL3sgMHDlitWrUsKCjIfY/RypUrrWHDhrxf74OQBNu4caMdOXLE9u7da9myZbMxY8aY2Z3nD7hcLnO5XO77BcLCwqxs2bIez0rC37u78/nkk08sT5489ueff5rZ/6b2PnDggLlcLlu/fr3HdkuXLrV48eI9cVNumt25Pytz5szWqlUr27lzp4WFhVlgYKA1atTI496VNm3aWOLEie3rr7/2XrH4R5MmTbKsWbO6bxT+/PPPLW7cuObn52cDBw50t5s/f7717NmToR738cUXX1iWLFncJwu2bNliLpfLUqdObb1797ajR4+627Zu3Zp+/Bf+etC4adMmS5AggUfwDA8Pt40bN1qBAgVszpw5//gaT7rz589b+fLlbfLkye5lZ8+etdGjR7snF7nr7bff5lmA/8KmTZssfvz4HkPpwsPDbdu2bVaoUCGbMWNGpCtzvF8j8/H2w2wRdW7duuX+/4iICPf/lypVSkFBQdq1a5fSp0+vpk2bSpKSJ0+u9u3b6/3331f+/PklSQcPHtTJkydVrVq1qC0+BnH2rfS/J7LXrFlTx44d05tvvilJihs3riTp2rVryps3r5ImTeqxXfr06XXw4EHlyZPnP685ukmTJo3mzZunvXv3avTo0QoNDdWMGTO0ZcsWffTRR9q+fbsk6bPPPlONGjXUoUMHffnll7px44aXK4f0v7+BiIgI3bx5U8eOHdPLL7+sUqVK6bvvvtOrr76qYcOGqWfPnhoyZIhGjhwpSapfv77effdd+fr6Kjw83Ju/gtc1bdpUo0ePdn9/8+ZN+fj4qEePHipatKi++eYbVa5cWV988YV69+6t9957T1OmTNH+/fslSZMmTaIfH5GZuffbU6ZM0enTp5UjRw5ly5ZNS5Yscb+/fXx8lD9/fl29elXHjh2L9Dp3XwN3xI0bV0eOHNGpU6fcy1KlSqXg4GBlzpxZe/bscS8fMGCA4sSJo9u3b3uj1Bhr+fLlkqQCBQoob968WrJkifvYz8fHR3nz5tWNGzd09OhRuVwuj215v96Dt1Ma/nt/nflr/Pjx1qFDB+vRo4d99dVX7uWjR482l8tlx44ds4sXL1rdunWtY8eO7vV3zzIwI9L9Oc/EbNy40RYsWGC//PKLe/KFr7/+2vz9/a1Vq1a2YsUK27p1q9WsWdPKlCnDWZx72LZtmxUuXNhat25tf/75p61du9YyZsxoLVu29HjGSK1atSxz5sweNwHDO5zv4+PHj5uZ2YULF+zQoUN2/Phxy5Mnj33wwQdmdmeWpQQJEpjL5Xpih5Xey9mzZ23ChAmRHiR9+PBh++OPP+zkyZNWtGhRdz+eP3/eUqZMaX5+fh4PNcXDOXz4sMcEAdu3b7d06dLZpUuX7NatW9ayZUt75plnPK5cX7lyxUqWLGkTJ070VtnR3qVLl8zszuiJ4OBga968eaTJdlq3bm3PP/889x39Cxs2bLCsWbPapk2b7NatW9auXTsrXbq0x/2fV69etVKlSrG/fUCEpFjulVdesYoVK7rvK3rrrbcsYcKE1rx5cytatKjlzZvXmjRpYmZ3pjWtWLGiuVwuy5kzp+XLl8/jQ5qD+L/n3Ln37t3bgoKCLH369Fa4cGHr2LGjeyjMsmXLLCgoyDJmzGg5cuSwcuXKufuZPo7MGZQuXrzoEZScQ+/uN8Mdoo7z/du3b19zuVwe95F9//33lj9/fvcU3lu3brVmzZrZ/PnzGRr2//76PJJx48ZZ586dPZZt377d8uTJY+vWrTOzO0N2O3fubJMmTaIfH1Hnzp09JtYxu/NMmQwZMrjvNzp//rxVr17dSpYsac2aNbOPP/7YKlSoYPnz52do2H0MGTLEXnrpJTt8+LCZmc2bN88CAgKsX79+duTIETO7c+D+9NNPW8+ePb1Zaox37Ngxy5w5s40YMcLMzEJCQqxu3bpWrFgxa9iwob377rtWvnx53q8PgZAUy61bt85y5sxpzz//vC1atMhq1Kjhvhny2rVrNm3aNMubN6+1a9fOzO6c8fnyyy9txowZ7g9b/pgezjvvvGOBgYHufn7llVcsWbJk1qhRI/dVvQsXLti+ffts165d7gNL+vn+7nVFKWvWrNagQQP3bGicgfQuZ0Dq3Lmz+fv7W/LkyT1uHL575WjixIl2+vRpq1mzpjVv3tz9b/ek/w107tzZateubWZ3puC9cuWK9enTx3LkyGH9+vVzt1uxYoWlSJHCxowZY2vWrLHatWtb/fr13esJSg/v2rVrlidPHitatKj7Po6NGzda/vz5zex/783z58/b0KFDrUKFCvbMM89Y06ZN3Se56HdPvXr1snTp0tlnn33mcRJr2rRpljp1aitbtqxVrVrVypYta/ny5Xvi//4fxd19p/Pe50yZMrk/F69evWrvv/++1apVyypWrGgtW7bk/foQCEmx2N0/mp9//tmyZ8/ufkjp3elKze4MFfj444+tcOHC9ssvv0R6Df6IHs6xY8fs2WefdQ9jXLx4sSVOnNiaNWtmefLkscaNG99zaliuIP2zv15RWrlypeXPn9/j/Qzv69atmyVLlsx+/fVXq169usf0vhcuXLDu3btb/PjxLSgoyAoXLuz+wCbkmu3evdvdH5cvXzYzsxMnTtjQoUMtT5481qdPH3fbrl27WvLkyS1LlixWqlQp+vFfuNt3oaGhlitXLitUqJDt3LnTFixYcN+ZR83M41k+HOB7+u677yxt2rQeEwdcvHjRDhw4YGZ3Ziv94IMPrE2bNjZ48GB3/9GPD+evt1Ns377dSpUqZTNmzIjU9u5DkM3o5wdFSIql/nrQvWHDBsuVK5e5XC6bPXu2x7pff/3V/Pz8mCHsMVm6dKmdOXPGNm7caIGBge7p0jt06GCJEye2ypUr22+//eblKmOmbdu2WfHixa1Ro0Z26dIlCw0N9XZJcOjfv78lTJjQPWSpWrVq1r9/fzP738H7hQsXbMuWLbZw4UKuVt/H559/bsmSJXPvJ06ePGmDBw+2PHnyeAxJ+vnnn23nzp1cjf4X/vqw4mvXrlnOnDmtfPnyNnToUEuXLp29/vrr1qtXL/vggw9s+PDh1r59e/dzBZ3b4n+mTJli5cqVM7M7gWjw4MGWPXt2S58+vbVt29Y9y6sTJ2Ufzvfff28ul8tatmzpcS9i9+7dLWvWrPd9EDfv1wfHVBax1N1ZSlasWKGzZ8/qqaee0pdffqkcOXJo8uTJWrNmjbtt4sSJFRQUJF9fX2+VGyPdb9aoSpUqKXXq1Jo3b54qVKigtm3bSpIyZcqkwoULq2TJksqQIUNUlhprFClSRJ988olOnz6t0NBQ+fv7e7skODzzzDPavHmzihYtKknKmzevLly4IEnumZROnTqlYsWKqVatWu7Z1+LEieO1mqMD52yAkpQ5c2YVKFBAVatW1fHjx5U+fXq1atVKL774ohYtWqTevXtLkooXL66CBQvKx8eHfnxEdz8rV65cqf379ytBggTavn27zpw5o/79+ytr1qw6evSo9u7dq02bNmnjxo06deqUChcu7H6Nv84S9iS7O8NounTptGbNGgUHB6tGjRo6cOCA3njjDfXt21fffvutTp48GWlbjkEeXFhYmKpXr65vvvlGLpdLQ4cOValSpTR16lQ1aNBA2bNn1xdffHHPbXm/PgRvpzT8d3766SfLmTOnde3a1T0j3caNGy179uz21FNP2dChQ23OnDlWp04dy5MnD2dxHoLzSt28efNsypQp9vnnn7tn8TG7c+XomWeecfd9gwYNbOLEiZHGEOPhOYcNwDv+7v179z3eu3dv93CliIgIe+aZZyx37tz/uP2TyjkRyYYNG6xSpUqWPXt2jytKQ4cOtRQpUtj48eO9VGXss3nzZkuVKpV1797dPXzp+vXrVrBgQStYsKDt2LHjntvxmelp+vTplidPHvfELLNnz7Y2bdrY559/bn/88YeZ3bmSXKRIkUjPBcSDu3sv+d1+vnz5sp09e9batWtnVatWtUSJEpmfn581a9bMy5XGfISkWK5///5WpkwZ6969u509e9bM7nwg5M6d21wul9WvX99ef/119zANdvp/r0GDBtarVy/39927d7fEiRNb4cKFzc/Pz5555hmbNWuWmd35wChatKgVLVrUihQpYrlz53b3M5e7EZM537+ffPKJ9ezZ015//XW7evWqmf0vAH300UfukFSzZk3Lly+fx30cTzpnUNy0aZO5XC6P6XrXr18fKSgdP37cpk6dyr76Mfvggw8sb9681rNnT/d9M3eH3hUrVsw2bNhAsP8H8+fPt6eeesqefvpp9/HG3RNat27dsqtXr1r16tXt6aefpi//hfnz51upUqXs6aefjvRIljNnzthnn31m5cuXt8SJE9v8+fO9U2QsQUiKJZwHLX/98Bw4cKCVKlXKunfv7v6D2rFjhwUEBNg777xz3+3g6fbt2/b+++9b3LhxbciQIXb27FkrXLiw/fzzzxYaGmpnz561GjVq2DPPPGPLly83M7NJkyZZ//79rVevXgRRxArOg5s+ffpY8uTJrUaNGhYUFGSZMmXymABm+fLlljdvXitVqpRlzZrVfYM898547rM//fRT6969u/n4+Fj8+PFt+vTp7nX3CkrsSx6ds9//ek/jhx9+aLly5bKePXu6J9gJDQ21pEmTWsuWLaO0zpgoIiLCvv/+eytbtqyVLl3afbxx9epVGzlypJUrV86KFSvGIy/+JWc/lylTxt3Pzv48cuSINWrUyPr27eveBg+PkBTLTJkyxQYPHhxp5z9w4EDLmTOnvfHGG+4Hm+7Zs8f9Icsf0IMJCwuz8ePHm4+PjzVu3NgaNmxooaGh7v47c+aMlSlTxqpXr37P7Tk4RGxx6dIla9u2rW3bts0iIiLs5MmT9uyzz1qGDBls3759Zmb2448/msvlskKFChGQ7qN3796WNm1amzJlio0cOdLq1KljiRIlssmTJ7vbrF+/3ipWrGg5cuRwz+bIAea/M3r0aBs8eHCkM/EjR460VKlS2euvv+4OSjdu3CCQ3sfcuXM9jjciIiJs0aJFVrZsWXv66aftwoULZnbnQepvvPEGs9g9ogftZ+eV+vfee8/y5MnDBEf/AiEpFgkPD7fg4GArWrSojRw5MtIfRs2aNS0wMDDSzDLs/B9OWFiYffrppxYQEGDZsmWzkJAQM/vfNLJr1qyxePHi2e7duwmfiJUmTpxoiRMntrJly7ofCGlmdvbsWatatapHUJo1axYHRvdx4sQJK1CggMcQu19//dVee+01S5AggXsa34iICNu4caNVqVLFnn32WfewRjy4v4bKV1991QIDA+3DDz+MFJTatGljadOmtQ4dOnjMRMpnpae1a9da8eLFrV69epGml541a5alSJHCqlev7h56dxf9+HAepJ+rVavm7ue7xx39+vWzYsWKuY9R8PCY3S4GuzsT0l0+Pj6aMmWKihcvrlmzZumTTz5RaGioe32BAgWUJk0aJUqUSAEBAe7lzCjz98zM47/x4sVTkyZNNHLkSJ04cULDhw+XJMWNG1fSnVnvMmbMqAQJEjCLDGKlEiVKqHjx4tq5c6d7lseIiAilSpVKM2bMUP78+VWgQAEdO3ZMjRs3Vpw4cXT79m1mX7uHo0ePeuyns2XLpo4dOypr1qxq0aKFZsyYIZfLpZIlS6phw4YKDQ3VtWvXvFhxzHR3FrtJkybpxIkTGj16tF588UWNGjVK06dP17lz59xtAwMDFRgYqBs3bihjxozu5XxW/s/cuXM1YcIENWvWTOfOndNLL73kntkuTpw4qlGjhjJkyKC1a9eqb9++HtvSjw/uQft5/fr16tevn6Q7xypnz57Vpk2bNGHCBCVOnNibv0LM5uWQhkd096qF2Z1Z7DZv3my7du0yszs3SrZp08ZKlSplI0aMsIsXL1pERIS9+OKLNnv2bGZXewjOPrp48aJdvHjR/X1oaKiNGzfOfH19rWvXrrZ27Vrbs2eP1ahRw5566in6F7HCvd7Ht2/ftt27d1vRokUtf/787lkd7+5bTp8+bV27duWMscO9+vHGjRvWsGFDa9mypZ04ccJjXfPmze3pp5+29OnT2+LFi83szrDpwMBA99AaPLz06dNbvXr13N/36NHDMmfObO+99557ZrtGjRrZkiVL3O9nRgRENnDgQCtbtqzt2rXLZsyYYUWKFLGGDRu6158/f95eeuklW7x4MZ+F/8K/6Wdmgf33XGb/f3ocMcJLL72k559/Xs8995wkqWfPnpo6darix48vPz8/DRo0yH2moXv37tq8ebP+/PNPpUyZUleuXNGePXvk6+uriIgI95k13Juzj95991198803un79ujJmzKgvv/xSCRIk0K1btzRp0iT16NFD169fV7du3XT48GHNmTNHcePGpZ8Roznfv/v27ZOvr6/ix4+vzJkzy8y0f/9+vfjii4qIiNDatWuVJEmSSO/58PDwJ/7MsbNPfvvtN92+fVvZsmWTJE2YMEEjR45U06ZN1apVK2XIkEFXrlxRy5YtVatWLc2fP18ZMmTQuHHjtGzZMqVOnVqFChXy5q8TY9xr/7t+/Xq1bNlSPXr0UIcOHSRJ/fr107fffqurV68qUaJEun37tnbv3q04ceKwD/+LP//8U8mTJ5d05zld6dOn1zfffKOZM2fq/ffflyS1bdtWX331leLHj6/vv//e/RyvJ30/8DAetZ+d71czYzTLv+XdjIaHcfr0aWvYsKElT57cli5daidOnLAcOXLY5s2bbfny5fbGG2+Yy+Wyzz77zMzu3Dszf/58e+utt+ztt99mRqSH4Dxz2LdvX0ubNq2NHTvWfvjhB0uTJo1VqlTJfdbxxo0bNnXqVHO5XDZx4kT3dtx/gZjM+TcwaNAgy5Url2XPnt1SpkzpMa3s3r17rXDhwla4cGGPK62IrE+fPpYlSxZLkyaN1apVy/2ckxEjRli+fPmsdOnS1qJFCytRooQVLVrUzMzatm1r5cqV42rGvzBmzBhbtmyZnTt3zm7fvm2dO3e2Ro0aedxPt2jRIhszZoyNGDGCz8r7GDp0qNWsWdO+++47MzPbv3+/5ciRw8aMGWNmZitXrrQaNWpY8eLFrX79+sxi94jo5+iDkBTDHDlyxNq2bWvJkiWznj17Wp8+fdzrzp8/b/369TOXy2WTJk265/bs9P+ec/piM7MffvjBChQoYD/99JOZmS1evNgSJ05sadKksXz58rlnP7p+/brNnz+fYIRYZ9CgQZYmTRpbsmSJnT592l544QWLHz++ffrpp+42+/bts3Tp0tlLL73kxUqjH+dBy8yZMy1z5sw2a9YsmzVrluXOndsKFSrk3ocsWrTIBg4caLVr17auXbvajRs3zMzsxRdftJdfftljiDUe3K5du8zlclmmTJmsU6dOtnr1ajtx4oSlSZPmbx/Iy2elp9u3b1vDhg3N5XJZokSJrG/fvrZt2zbr27evvfjii+73sZnZuXPn3KGez8SHQz9HL4SkGML5B7B48WLr37+/+fv7W9OmTT3aXbhwwfr162e+vr42duzYqC4zRmvZsmWkZwosW7bMRo4caWZmS5YscT/p/tixY5YqVSqrXLmyexavu9hZIbbYvn27VapUyZYsWWJmZt98840lS5bMatasaT4+PjZhwgR326NHj3JgeR8LFiywTz75xKO//vzzT8uXL58VLFjQfVXa7H/B6uzZs9a3b19LliyZ7d27N8prjqn+esXtxo0b1rx5c8uUKZN9+umnlj59eps0aZL16dPHkiRJYnv27Lnndojsxx9/tObNm9snn3xiFSpUsA4dOlijRo0sS5Ys9sknn0RqT58+Gvo5+iAkxQDOs5EjRoyw7Nmz21dffWUvv/yy+fr62qJFizzaX7hwwTp37mxlypThj+cBhYSE2MaNG91na//44w/3uuPHj9v169etYsWKNmDAADO7c4BTokQJc7lc9uKLL3qlZuBx++twjcOHD9vHH39st2/ftpUrV1q6dOls9OjRZmZWo0YNS5gwoX344Yce2xCUPJ07d84SJEhgLpfL3nrrLTP730HNxYsXLX/+/FasWDHbuXOne5sLFy5Yq1atLE+ePLZ9+3ZvlB3jLV++3LZu3WpmdwJntmzZbNSoUbZz504rUqSIPf/88+Zyuaxp06Z2+fJlL1cbfY0cOdI++OADM7uzf2jVqpW1bt3abt68adOmTbO2bduay+Uyl8tlu3fv9nK1MRf9HD0RkqI55wFH586dzeVymb+/v61bt87Onz9vbdq0sYCAAPfsR3eFhIQwM88Devnll61Xr17u7ydOnGh169a1TZs2uZedPHnSsmfPbgsXLjQzsytXrljz5s1t//79jANGrOB8H999QKyZuWeua9mypbVv3959pbRt27ZWsGBBe/rpp9nHONyrL/bu3Wt58uSxMmXK2KlTpzzaXbx40VKlSmUtW7b02Oa3336zkydP/vcFx0Lbtm2zkiVLWsGCBW3atGlmdme4Y7Nmzez06dN26tQpGzNmjGXJksUqV67M+/c+bt68aUOGDDFfX18LDg62ZcuW2e3bt61o0aL27rvvutt069bNqlatygmSR0Q/R1+EpBiiR48eFhAQYAcPHrRKlSrZ1KlTzczswIED1q5dO0uWLJl7SIwTO/9/tmDBAvcVpOvXr9vixYutQIEC1qJFC/eZyIiICMudO7dVqFDBvvjiC6tYsaKVKlXKfWDJTgsxmTMgDRgwwMqUKeN+wGl4eLiFhoZasWLF3PdA3rp1y5577jlbu3YtJ2McnP1469Ytj6G3u3fvtnTp0ln16tXdDy+922dXrlzx2IfQlw/nXieqNm/ebEOGDDEfHx975ZVXbNSoUfbiiy+6Q1NYWJidO3fOvS19fn979uyx5557zkqWLGmtWrWyL774wl544QX356PZ//qPz8JHRz9HP4SkGODurHU7duwwM7PixYvb4MGD3euPHDliHTp0MJfLZevWrfNWmTHOXz8Up06dajVq1LA///zTVqxYYUWKFLFmzZq5ryjt2LHD8ufPb0WKFLFnn32WGWUQ6/Tu3dtSpEhhy5cvd1/xuKtv374WN25c69ChgxUrVswKFizo/qDmANNzP/D+++9bcHCwFS1a1N577z3bvHmzmd0JSmnTprUaNWrY+fPnzcyz7zjweXjOft+4caN9//33du3aNfeyn376yapUqWL169e35MmTW2BgoO3fv9/jNej3f3bu3DmbN2+eFS9e3OLFi2cpUqTwOA4xYz/wONDP0QuT/8cAVapU0a5du9zPxsiYMaP7icuSFBQUpGeeeUbjxo1TyZIlvVVmjPPX5weEhobq4sWLevXVV1WkSBG999572rt3r0aPHq3t27erUKFC2rFjhxYuXKilS5cqbty4un37Ns/QQKywdetWLViwQN99950qV66shAkT6siRI/r000914sQJDRo0SH369NHvv/+uwoULa8uWLfL19VV4eDjP4pDc+4E+ffpo2LBhyp07t/Lly6c5c+aoe/fu+vHHH5U/f34tX75ce/bsUY0aNXT58mWPvuM5Mg/vbr+//vrrql+/vho1aqSSJUtq+vTpCgkJ0TPPPKOJEyeqfv36Klq0qE6dOqWpU6d6vAb9/s9Spkyp5557Tj///LN69uyp0NBQrVixwqMN+4F/j36OZryd0vDg7p7t6tSpk/uJy7du3bI6depYvXr1mAryMZg6dao9/fTT1rhxY/vzzz9t+fLlVrRoUWvevHmkq3RcQUJM9tf37+bNmy1lypS2fft227Nnj7322muWI0cOS5s2raVJk8ZOnDhhZmahoaHubdjXeNq1a5flypXLfvzxR/eyH3/80YKDg61KlSr266+/mtmdq9L16tVjH/IvOM+m//DDD1akSBH78ccf7ciRI9aoUSMrUqSIjRkzxkJCQszszudnaGioDRkyhPftI3L2+aZNm7iS/B+hn6MPToHHIHfPdiVIkEBnzpyRJNWvX1/79u3TnDlz5HK5ZGaKEyeON8uMkcxMktSiRQu1adNGv//+uzp27KiiRYvqvffe048//hjpbA5XkBCT3X3/Hjx4ULdu3VLWrFlVqlQp1alTR6VLl9bNmzf19ttv67ffflPcuHE1b948SZK/v78ksa+RFBER4fH97du3debMGY9+qVixolq0aKFDhw7p5MmTkqRChQppwYIF8vHxifQaeDB3z6bPnj1bS5cuVe3atVWxYkUFBQXpq6++UqFChfTZZ5/p888/19WrV+Xr6yt/f3/169dPceLE0e3bt738G8Q8d48xJKlkyZJcSf6P0M/RB0d5McjdP5oiRYrIzFSpUiUdOHBA+/fvdw/94o/o0Th3Si1btvQISsWKFdPChQvVu3dvL1cJ/HvOg/IZM2aoYcOGWrx4sVKkSKHRo0frvffe04IFCzRq1CgFBwfr1q1bSp8+vQIDAz1e50nf10RERLiD5rx58/Tbb78pICBAadKk0W+//Sbpf/vs6tWry8/PT2vWrIn0OpxseTh3+zQiIkK3b9/We++9p5EjR2rPnj0e7aZMmaIiRYpo8uTJGjdunMcQdUlPfMB/VH/9u2eo4n+Dfo4e2DvHIHf/aJInT661a9fq3Llz2rdvnzsgsdP/d/4alFq3bq2TJ0/q5ZdfVkBAgLufgZjKeWC/YMECHTt2THv37tVbb72l77//XpkyZVJwcLAqVaokM9PRo0cVHBys27dv67nnnvNy9dGHmbn7sW/fvurSpYu+++47Zc2aVU899ZR69Oih9evXu9tfvHhRCRIkUMaMGb1Vcqxx93Pw7NmzihMnjn766SfVq1dPO3fu1OzZs3Xz5k1328mTJytDhgzat2+f/Pz8vFUygBjKZXePChFjXLt2TV988YXatGnjHjZAQHp8zMz9QTxt2jRNnjxZpUuX1ltvvaV48eI98WfQEfP17dtXEydO1KBBg3Tt2jVNnjxZ8ePH15AhQ1SzZk35+vpq6tSpmjlzpq5du6ZVq1Ypbty4Cg8P54ymw+DBg/Xxxx9r8eLFypEjh5ImTSpJaty4sVavXq0XX3xRKVOm1KpVq3T69Glt27aNffVj8Pnnn2vWrFl68803VaJECV2/fl1169bVpUuX1KdPH9WpU0dx48Z1t797csC5bweAf0JIiuEISP8N54dpz549tXHjRq1YsULx4sXzcmXAv3Po0CFVrlxZH3/8serXry9JCgkJ0bPPPqurV6/q3XffVa1atbR9+3bt2bNHL774onx9fdnX/MWff/6pxo0bq2XLlmratKl+//13HTx4ULNmzVLlypX1zTffKDQ0VGfPnlX27Nn12WefETQfkylTpmjChAnKli2bunbtquLFiys0NFR169ZVSEiI+vTpo9q1a98zKAHAg+ITL4bjoOW/cXfoncvlUqJEifTHH3/o+vXrhCTEeHevht4dfhQWFqYkSZLohx9+UM6cOTV8+HD5+PioRo0aKlKkiCQpPDycfc1fuFwu7du3T/v379dPP/2kTz75REePHlV4eLgWLVqk/v37q127dgoLC1OCBAkkcVLrUdwr3LRq1Ur+/v4aO3asPvjgA/Xo0UPFixfXt99+q+eee05du3ZV8uTJVb58efc2BCQAD4u9BnAfd4NS3rx5NW/ePAUEBHi7JOCh3GvmtGTJksnHx0dLliyRJPn5+enWrVtKnDix8uXLp2PHjmnw4MH69ddf3a/BlY/IkiVLprfffluffPKJ6tSpo8yZM2vo0KHasmWLypcvrw0bNsjX19cdkJgN8NHcDTfLli3T4cOH3cuDg4PVsWNHnTx5Uu+//7527typBAkSaN68eXruuef09NNPe6tkALEEw+0AIBZynoE/cuSIkidPLh8fHyVJkkRz5sxRs2bNNHDgQPXr10/SnYP41q1bq127dmrUqJHq16+vMWPGePNXiBGOHz+usLAw5ciRQ9Kdfq9ataqeeuopDRkyxMvVxVzO9++OHTtUt25d1atXTz169FCWLFnc7aZOnaouXbqodu3aevXVV1WmTBn3OoY2Avg3CEkAEIv1799fX3/9tW7evKlatWrp5ZdfVt68efXhhx+qV69eqlGjhjJlyqSdO3fq/Pnz2r9/v9q2bavTp09r4cKF3i4/xrh69ap27NihESNG6LfffmOShn/BGZC+/fZblStXTtOnT9fnn3+uMmXKqFu3bh5BqXDhwrpw4YLatm2rQYMGMUEDgMeC4XYAEIs4z3vNnz9fkyZN0vDhw9W4cWPt379fXbp00d69e9WtWzetWLFCZqaTJ08qW7Zs2rVrlyTpzJkzypw5s7d+hRjHzLRlyxaNGDFCt27d0tatWxUnThyFh4d7u7QY56/Tq7dv316zZs1Sly5d1KRJE/30008aNWqUjh07Jkk6ffq0SpQooSFDhmjAgAGSeIYXgMeDK0kAEAstXrxYP/74o3LlyqV27dpJunNWfty4cbp586bef/99FSlSxGMygdDQUL355puaNm2aVq9erdy5c3vzV4hRwsLCtG/fPhUqVEg+Pj5M0vAv3W969XHjxunzzz9XsmTJVKlSJf3www+SpCVLlsjlcjGLHYDHhj0JAMQy27dv14ABAzR58mSPyRvq1q2rjh07ys/PT7169dLGjRvdB/IHDx7UO++8o7lz52rp0qUEpIfk5+enIkWKyMfHRxEREQSkf+HPP/90XzEqUaKErl27ppUrV6pDhw5KmTKlateurWTJkmnq1KlKkCCBFi5c6J5oh4AE4HHhShIAxEKTJk3SqFGjlChRIs2YMUNZs2Z1r1u4cKEGDx6sUqVK6eOPP5Yk3bp1S/v371fKlCkVGBjorbIBXbx4Ufnz51erVq1UtWpV9/TqEREROnnypAYMGKAOHTro8uXLSpYsmVwuF1fuADx2hCQAiMH+bnjR5MmTNXnyZGXKlElDhw5VUFCQe926detUunRp+fj4cKM7op1JkyapZ8+eCg8P18svv6xnn31WVapUUbNmzeTr66tp06a52zLEDsB/gZAEADGU8+Bwzpw52rNnj1KmTKlChQqpXLlykqRPP/1UM2bMUIYMGTRs2DCPWcH++hpAdML06gC8iZAEADGQ8+pPr169NGPGDBUpUkShoaG6evWqOnfurJdeeknSnaA0a9Ys+fn5acqUKUqXLp03SwceCtOrA/AGTh8CQAwTERHhDkhjx47V7Nmz9fXXX2vhwoV64YUXtHPnTr355puaMGGCJKlDhw6qU6eOsmXLpjRp0nizdOChML06AG/hShIAxCDOK0ghISHq27evsmTJotdff13ffvutmjdvrtdee02HDh3STz/9pOHDh7uvKN3dliF2iEmYXh2ANxCSACCGWLlypf744w81bdpUHTp0ULJkyfTaa6/p+vXrCg8PV82aNdWpUyd17dpV8+fPV5MmTRQ3blxNmTJFL7zwgiQxSQNiNAI+gKjCqRgAiAGuXLmi4cOH6+bNm/rqq6+0evVqrVmzxn1/0RdffKHkyZOrVatWkqSECROqVq1aqlGjhp577jn36xCQEJMRkABEFfY2ABADJE6cWLNmzdLp06e1cOFC9e3bVwULFnSvjxs3ro4fP641a9YoNDRUo0ePVpYsWdSmTRv5+vpyDwcAAA+BK0kAEEP4+Pi4J19YsWKFMmTIoKZNm0qS8ubNq3Llyql58+ZKmjSpEiZMqHnz5snlcsnM5Ovr6+XqAQCIObgnCQBimNOnT6tNmza6fv262rRp4w5KBw4c0C+//KIrV66oSZMm8vX15SZ3AAAeAcPtACCGSZs2rcaMGaMECRJo2rRpmjx5ssLDw/XKK69o9+7datasmXuIHQEJAICHx5UkAIihjh49qtdff1379+/XjRs3lDBhQm3dulXx4sXzdmkAAMRohCQAiMFOnTqlrVu36syZM2rRooXixInDEDsAAP4lQhIAxCLh4eFM0gAAwL9ESAIAAAAAByZuAAAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAOB1x44dk8vl0o4dO7xdCgAAhCQAAP5JhQoV1LVrV2+XAQCIIoQkAECsdfPmTW+X4CG61QMAuDdCEgAgykREROjdd99V9uzZ5efnp0yZMmno0KHu9UeOHFHFihWVIEECFSpUSBs2bHCvu3Dhgpo0aaL06dMrQYIEKlCggL788kuP169QoYJeffVVde3aVSlTplS1atUkSSNHjlSBAgWUMGFCZcyYUa+88oquXr3qse26detUoUIFJUiQQMmSJVO1atV08eJFtWzZUqtXr9ZHH30kl8sll8ulY8eOSZL27NmjGjVqKFGiREqTJo1eeuklnT9//h/rAQBEb4QkAECU6dOnj9555x0NGDBA+/bt08yZM5UmTRr3+n79+un111/Xjh07lDNnTjVp0kS3b9+WJN24cUPFihXTokWLtGfPHrVv314vvfSSNm/e7PEzpk2bpnjx4mndunUaP368JMnHx0cff/yx9u7dq2nTpunHH39Ur1693Nvs2LFDlStXVt68ebVhwwatXbtWderUUXh4uD766COVLl1a7dq106lTp3Tq1CllzJhRly5dUqVKlVSkSBFt2bJFS5Ys0ZkzZ9SoUaN/rAcAEL25zMy8XQQAIPa7cuWKUqVKpTFjxqht27Ye644dO6agoCB99tlnatOmjSRp3759ypcvn/bv36/cuXPf8zVr166t3Llz6/3335d058pNSEiItm3b9re1zJ07Vy+//LL7qs+LL76o48ePa+3atfdsX6FCBRUuXFijRo1yLxsyZIjWrFmjpUuXupedPHlSGTNm1IEDB5QzZ84HrgcAEL3E8XYBAIAnw/79+xUWFqbKlSvft03BggXd/58uXTpJ0tmzZ5U7d26Fh4dr2LBhmj17tn7//XfdvHlTYWFhSpAggcdrFCtWLNLrLl++XMOHD9cvv/yikJAQ3b59Wzdu3FBoaKgSJEigHTt2qGHDhg/1++zcuVMrV65UokSJIq07fPiwcubMed96AADRGyEJABAl/P39/7FN3Lhx3f/vcrkk3bmPSZLee+89ffTRRxo1apT7/qKuXbtGmgwhYcKEHt8fO3ZMtWvXVseOHTV06FAlT55ca9euVZs2bXTz5k0lSJDggWr7q6tXr6pOnToaMWJEpHV3A9696gEARH/ckwQAiBI5cuSQv7+/VqxY8Ujbr1u3TvXq1VOzZs1UqFAhZc2aVQcPHvzH7bZu3aqIiAh98MEHeuqpp5QzZ0798ccfHm0KFiz4t3XFixdP4eHhHsuKFi2qvXv3KkuWLMqePbvHF8EIAGI2QhIAIErEjx9fb7zxhnr16qXp06fr8OHD2rhxoyZNmvRA2+fIkUPLli3T+vXrtX//fnXo0EFnzpz5x+2yZ8+uW7duafTo0Tpy5Ig+//zzSBMo9OnTRz///LNeeeUV7dq1S7/88ovGjRvnvmcpS5Ys2rRpk44dO6bz588rIiJCnTp10p9//qkmTZro559/1uHDh7V06VK1atUqUqACAMQshCQAQJQZMGCAevTooYEDBypPnjxq3Lixzp49+0Db9u/fX0WLFlW1atVUoUIFpU2bVvXr1//H7QoVKqSRI0dqxIgRyp8/v2bMmKHhw4d7tMmZM6d++OEH7dy5UyVLllTp0qX1zTffKE6cO6PSX3/9dfn6+ipv3rxKlSqVjh8/rsDAQK1bt07h4eGqWrWqChQooK5duypp0qTy8eHjFQBiMma3AwAAAAAHTnUBAAAAgAMhCQAAAAAcCEkAAAAA4EBIAgAAAAAHQhIAAAAAOBCSAAAAAMCBkAQAAAAADoQkAAAAAHAgJAEAAACAAyEJAAAAABwISQAAAADgQEgCAAAAAIf/A0reZjRlmH/DAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 假设 ds 是已定义的数据集\n",
    "#print(ds[0])\n",
    "name = defaultdict(int)\n",
    "for pt in ds:\n",
    "    string = pt[0]['prompt'].split()[1].replace(\"'\", '').replace(\",\", \"\")\n",
    "    string = ''.join(lazy_pinyin(string))\n",
    "    name[string] += 1\n",
    "\n",
    "print(name)\n",
    "keys = list(name.keys())\n",
    "values = list(name.values())\n",
    "\n",
    "# 创建figure和axes\n",
    "fig, ax = plt.subplots(figsize=(10, 6))\n",
    "\n",
    "# 绘制条形图\n",
    "bars = ax.bar(keys, values)\n",
    "\n",
    "# 添加标签和标题\n",
    "ax.set_xlabel('character')\n",
    "ax.set_ylabel('count')\n",
    "ax.set_title('character stats')\n",
    "\n",
    "# 调整x轴标签倾斜角度\n",
    "plt.xticks(rotation=45)\n",
    "\n",
    "# 在每个柱子上显示具体数值\n",
    "for bar in bars:\n",
    "    yval = bar.get_height()\n",
    "    ax.text(bar.get_x() + bar.get_width()/2, yval, int(yval), ha='center', va='bottom')\n",
    "\n",
    "# 显示图形\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.2 训练数据导入/上传到bos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bos_bucket_name = \"your_bos_bucket\"\n",
    "bos_bucket_file_path = \"/your_path/\"\n",
    "qianfan_dataset_name = \"your_ds_name\"\n",
    "\n",
    "# 创建千帆数据集，并上传保存\n",
    "qianfan_data_source = QianfanDataSource.create_bare_dataset(\n",
    "    name=qianfan_dataset_name,\n",
    "    template_type=console_consts.DataTemplateType.NonSortedConversation,\n",
    "    storage_type=console_consts.DataStorageType.PrivateBos,\n",
    "    storage_id=bos_bucket_name,\n",
    "    storage_path=bos_bucket_file_path,\n",
    ")\n",
    "qf_ds = ds.save(qianfan_data_source, should_overwrite_existed_file=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 多轮训练实验"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们基于ERNIE Speed模型共计实验了6次，参数和训练方法的配置如下。其中，实验1和实验4使用LoRA和全量更新中的平台默认参数："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "| | 实验1 (默认参数) | 实验2 | 实验3 | 实验4 (默认参数) | 实验5 | 实验6 |\n",
    "|-|-|-|-|-|-|-|\n",
    "| 训练方法 | LoRA |  LoRA |  LoRA | 全量更新 | 全量更新 | 全量更新 |\n",
    "| Epoch | 3 | 3 | 6 | 3 | 3 | 6 |  \n",
    "| Learning Rate | 3e-4 | 6e-4 | 3e-4 | 3e-5 | 6e-5 | 3e-5 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 实验1: Epoch = 3, Learning rate = 3e-4, Lora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer1 = LLMFinetune(\n",
    "    train_type=\"ERNIE-Speed\",\n",
    "    train_config=TrainConfig(\n",
    "        epoch=3,\n",
    "        learning_rate=3e-4,\n",
    "        #max_seq_len=4096,\n",
    "        peft_type=PeftType.LoRA,\n",
    "        #logging_steps=1,\n",
    "        #warmup_ratio=0.10,\n",
    "        #weight_decay=0.0100,\n",
    "        #lora_rank=8,\n",
    "        #lora_all_linear=\"True\",\n",
    "    ),\n",
    "    dataset=qf_ds\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer1.run()\n",
    "print(trainer1.result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 对微调结果进行评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "加载评估集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 16:10:57] dataset.py:407 [t:8149425664]: no data source was provided, construct\n",
      "[INFO] [06-12 16:10:57] dataset.py:281 [t:8149425664]: construct a qianfan data source from existed id: ds-bimjvfatbnard1we, with args: {'input_columns': ['prompt'], 'reference_column': 'response'}\n",
      "[INFO] [06-12 16:10:58] dataset_utils.py:305 [t:8149425664]: list qianfan dataset data by 0\n",
      "[INFO] [06-12 16:10:59] dataset_utils.py:327 [t:8149425664]: received dataset list from qianfan dataset\n",
      "[INFO] [06-12 16:10:59] dataset_utils.py:335 [t:8149425664]: retrieve single entity from https://bj.bcebos.com/easydata/_system_/dataset/ds-bimjvfatbnard1we/texts/data/raw_34e4c75f9f5d4d126148389de45a84caa5a7521ede6acb25f99b0ac26ad176c4_4d486c4cd19f425bb56bdb26a0f3e9ae?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-06-12T08%3A10%3A59Z%2F7200%2Fhost%2F53a3bd78e3b64903e4e518555a258c93305865b0f5caf2fb8115675c5cbf281a in try 0\n",
      "[INFO] [06-12 16:11:00] dataset_utils.py:349 [t:8149425664]: retrieve single entity from https://bj.bcebos.com/easydata/_system_/dataset/ds-bimjvfatbnard1we/texts/data/raw_34e4c75f9f5d4d126148389de45a84caa5a7521ede6acb25f99b0ac26ad176c4_4d486c4cd19f425bb56bdb26a0f3e9ae?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-06-12T08%3A10%3A59Z%2F7200%2Fhost%2F53a3bd78e3b64903e4e518555a258c93305865b0f5caf2fb8115675c5cbf281a succeeded, with content: [{\"prompt\": \"{'姓名': '孙悟空', '性别': '男', '物种': '石猴', '年龄': '在《西游记》中年龄未明确，实际年龄未知', '工作': '保护唐僧西天取经', '昵称': '齐天大圣、美猴王、斗战胜佛、行者、弼马温', '身高': '不满四尺（变化多端）', '生肖': '猴', '居住地': '花果山水帘洞', '爱好': '战斗、破坏、捣乱、吃桃', '学历': '拜须菩提祖师为师学艺', '智商': '非常聪明', '情商': '高，善于交际', '其他': '身怀七十二变、筋斗云等法术，拥有如意金箍棒', '经典台词': ['俺老孙来也！', '你这泼猴！', '妖怪哪里走！'], '口头禅': ['师傅'], '人物经历': '孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。', '人物关系': {'师父': '唐三藏', '师弟': ['猪八戒', '沙僧', '白龙马'], '结拜兄弟': ['牛魔王'], '嫂子': ['铁扇公主'], '侄子': '红孩儿', '敌人': ['白骨精', '黄袍怪', '红孩儿（后成徒弟）']}, '喜欢的事情/东西': '战斗、破坏、捣乱、吃桃', '不喜欢的事情/东西': '受束缚、被欺骗', '人物性格': '聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念', '外貌描述': '尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。', '服饰': '白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履', '武器': '如意金箍棒', '特殊能力': '七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢', '成就': '大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛', '封号': '美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨', '法宝': {'瞌睡虫': '孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。', '救命毫毛': '共三根。观音所赠，能协助孙悟空渡过难关。', '骷髅头': '月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。'}, '象征意义': {'思想意识': '孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。', '人心与道心': '孙悟空象征人心、道心、天心，是人类心灵最完美的象征。', '批判与颠覆': '孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。', '悲剧性': '孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。', '石崇拜文化': '孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。', '个人与团队': '孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。', '性格特点': '孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。'}}\\n现在请你扮演一个角色扮演专家。请你根据上述信息扮演孙悟空进行对话。\\n\\n好的！现在我来扮演孙悟空。\\n红孩儿：（朝上叩头）父王，孩儿拜揖。\\n孙悟空：（摆手摇头）孩儿免礼。\\n红孩儿：（躬身）孩儿不才，昨日获得一人，乃东土大唐和尚。特请父王同享唐僧之肉，寿延千纪。\\n孙悟空：（打了个失惊）我儿，是那个唐僧？\\n红孩儿：是往西天取经的人也。\\n孙悟空：我儿，可是孙行者师父么？\\n红孩儿：正是。\\n孙悟空：（摆手摇头）莫惹他！莫惹他！\\n红孩儿：（躬身）父王说那里话，长他人志气，灭孩儿的威风。\\n孙悟空：（笑道）我贤郎啊，你只知有三昧火赢得他，不知他有七十二般变化哩！\\n红孩儿：凭他怎么变化，我也认得，谅他决不敢进我门来。\\n孙悟空：我儿，你虽然认得他，他却不变大的，如狼犺大象，恐进不得你门；他若变作小的，你却难认。\\n红孩儿：凭他变甚小的，我这里每一层门上，有四五个小妖把守，他怎生得入！\\n孙悟空：你是不知，他会变苍蝇、蚊子、虼蚤，或是蜜蜂、蝴蝶并蟭蟟虫等项，又会变我模样，你却那里认得？\\n红孩儿：勿虑，他就是铁胆铜心，也不敢近我门来也。\\n孙悟空：既如此说，贤郎甚有手段，实是敌得他过，方来请我吃唐僧的肉，奈何我今日还不吃哩。\\n红孩儿：如何不吃？\\n孙悟空：我近来年老，你母亲常劝我作些善事。我想无甚作善，且持些斋戒。\\n红孩儿：不知父王是长斋，是月斋？\\n孙悟空：也不是长斋，也不是月斋，唤做雷斋，每月只该四日。\\n红孩儿：是那四日？\", \"response\": [[\"孙悟空：初一、十五并二十四、三十日。今日偏是二十四，正好持斋。\"]]}]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'entity_id': '34e4c75f9f5d4d126148389de45a84caa5a7521ede6acb25f99b0ac26ad176c4_4d486c4cd19f425bb56bdb26a0f3e9ae', 'entity_content': '[{\"prompt\": \"{\\'姓名\\': \\'孙悟空\\', \\'性别\\': \\'男\\', \\'物种\\': \\'石猴\\', \\'年龄\\': \\'在《西游记》中年龄未明确，实际年龄未知\\', \\'工作\\': \\'保护唐僧西天取经\\', \\'昵称\\': \\'齐天大圣、美猴王、斗战胜佛、行者、弼马温\\', \\'身高\\': \\'不满四尺（变化多端）\\', \\'生肖\\': \\'猴\\', \\'居住地\\': \\'花果山水帘洞\\', \\'爱好\\': \\'战斗、破坏、捣乱、吃桃\\', \\'学历\\': \\'拜须菩提祖师为师学艺\\', \\'智商\\': \\'非常聪明\\', \\'情商\\': \\'高，善于交际\\', \\'其他\\': \\'身怀七十二变、筋斗云等法术，拥有如意金箍棒\\', \\'经典台词\\': [\\'俺老孙来也！\\', \\'你这泼猴！\\', \\'妖怪哪里走！\\'], \\'口头禅\\': [\\'师傅\\'], \\'人物经历\\': \\'孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。\\', \\'人物关系\\': {\\'师父\\': \\'唐三藏\\', \\'师弟\\': [\\'猪八戒\\', \\'沙僧\\', \\'白龙马\\'], \\'结拜兄弟\\': [\\'牛魔王\\'], \\'嫂子\\': [\\'铁扇公主\\'], \\'侄子\\': \\'红孩儿\\', \\'敌人\\': [\\'白骨精\\', \\'黄袍怪\\', \\'红孩儿（后成徒弟）\\']}, \\'喜欢的事情/东西\\': \\'战斗、破坏、捣乱、吃桃\\', \\'不喜欢的事情/东西\\': \\'受束缚、被欺骗\\', \\'人物性格\\': \\'聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念\\', \\'外貌描述\\': \\'尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。\\', \\'服饰\\': \\'白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履\\', \\'武器\\': \\'如意金箍棒\\', \\'特殊能力\\': \\'七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢\\', \\'成就\\': \\'大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛\\', \\'封号\\': \\'美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨\\', \\'法宝\\': {\\'瞌睡虫\\': \\'孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。\\', \\'救命毫毛\\': \\'共三根。观音所赠，能协助孙悟空渡过难关。\\', \\'骷髅头\\': \\'月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。\\'}, \\'象征意义\\': {\\'思想意识\\': \\'孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。\\', \\'人心与道心\\': \\'孙悟空象征人心、道心、天心，是人类心灵最完美的象征。\\', \\'批判与颠覆\\': \\'孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。\\', \\'悲剧性\\': \\'孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。\\', \\'石崇拜文化\\': \\'孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。\\', \\'个人与团队\\': \\'孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。\\', \\'性格特点\\': \\'孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。\\'}}\\\\n现在请你扮演一个角色扮演专家。请你根据上述信息扮演孙悟空进行对话。\\\\n\\\\n好的！现在我来扮演孙悟空。\\\\n红孩儿：（朝上叩头）父王，孩儿拜揖。\\\\n孙悟空：（摆手摇头）孩儿免礼。\\\\n红孩儿：（躬身）孩儿不才，昨日获得一人，乃东土大唐和尚。特请父王同享唐僧之肉，寿延千纪。\\\\n孙悟空：（打了个失惊）我儿，是那个唐僧？\\\\n红孩儿：是往西天取经的人也。\\\\n孙悟空：我儿，可是孙行者师父么？\\\\n红孩儿：正是。\\\\n孙悟空：（摆手摇头）莫惹他！莫惹他！\\\\n红孩儿：（躬身）父王说那里话，长他人志气，灭孩儿的威风。\\\\n孙悟空：（笑道）我贤郎啊，你只知有三昧火赢得他，不知他有七十二般变化哩！\\\\n红孩儿：凭他怎么变化，我也认得，谅他决不敢进我门来。\\\\n孙悟空：我儿，你虽然认得他，他却不变大的，如狼犺大象，恐进不得你门；他若变作小的，你却难认。\\\\n红孩儿：凭他变甚小的，我这里每一层门上，有四五个小妖把守，他怎生得入！\\\\n孙悟空：你是不知，他会变苍蝇、蚊子、虼蚤，或是蜜蜂、蝴蝶并蟭蟟虫等项，又会变我模样，你却那里认得？\\\\n红孩儿：勿虑，他就是铁胆铜心，也不敢近我门来也。\\\\n孙悟空：既如此说，贤郎甚有手段，实是敌得他过，方来请我吃唐僧的肉，奈何我今日还不吃哩。\\\\n红孩儿：如何不吃？\\\\n孙悟空：我近来年老，你母亲常劝我作些善事。我想无甚作善，且持些斋戒。\\\\n红孩儿：不知父王是长斋，是月斋？\\\\n孙悟空：也不是长斋，也不是月斋，唤做雷斋，每月只该四日。\\\\n红孩儿：是那四日？\", \"response\": [[\"孙悟空：初一、十五并二十四、三十日。今日偏是二十四，正好持斋。\"]]}]'}]\n"
     ]
    }
   ],
   "source": [
    "from typing import Any, Dict\n",
    "\n",
    "from qianfan.dataset import Dataset\n",
    "\n",
    "\n",
    "# 导入平台上预置的测试集\n",
    "ds = Dataset.load(\n",
    "    qianfan_dataset_id=\"ds-bimjvfatbnard1we\",\n",
    "    organize_data_as_group=False,\n",
    "    input_columns=[\"prompt\"],\n",
    "    reference_column=\"response\",\n",
    ")\n",
    "\n",
    "print(ds[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "加载本地评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from roleplay_eval import RolePlayEvaluator\n",
    "from qianfan import ChatCompletion\n",
    "\n",
    "# 评估质量得分，按照客观得分模拟计算：\n",
    "local_evaluator = RolePlayEvaluator(\n",
    "    model=ChatCompletion(model=\"ERNIE-4.0-8K\"),\n",
    "    metric_name=\"accuracy\",\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对实验1进行评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 16:12:18] evaluation_manager.py:480 [t:8149425664]: start to inference in batch during evaluation\n",
      "[INFO] [06-12 16:12:19] dataset_utils.py:398 [t:6227210240]: start to create evaluation task in model\n",
      "[INFO] [06-12 16:12:20] dataset_utils.py:360 [t:6227210240]: start to polling status of evaluation task ame-vct2wvk322gy\n",
      "[INFO] [06-12 16:12:21] dataset_utils.py:367 [t:6227210240]: current eval_state: Pending\n",
      "[INFO] [06-12 16:12:52] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:13:23] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:13:53] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:14:24] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:14:55] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:15:25] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:15:56] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:16:27] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:16:58] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:17:28] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:17:59] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:18:30] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:19:01] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:19:31] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:20:02] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:20:33] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:21:04] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:21:35] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:22:05] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:22:36] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:23:07] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:23:38] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:24:09] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:24:39] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:25:10] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:25:41] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:26:12] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:26:43] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:27:14] dataset_utils.py:367 [t:6227210240]: current eval_state: DoingWithManualBegin\n",
      "[INFO] [06-12 16:27:14] dataset_utils.py:385 [t:6227210240]: get result dataset id ds-mxug9nd8t7azqifz\n",
      "[INFO] [06-12 16:27:14] dataset.py:407 [t:6227210240]: no data source was provided, construct\n",
      "[INFO] [06-12 16:27:14] dataset.py:281 [t:6227210240]: construct a qianfan data source from existed id: ds-mxug9nd8t7azqifz, with args: {'is_download_to_local': True}\n",
      "[WARNING] [06-12 16:27:15] baidu_qianfan.py:749 [t:6227210240]: parameter \"is_download_to_local\" has been set as deprecated\n",
      "[INFO] [06-12 16:27:16] baidu_qianfan.py:358 [t:6227210240]: no cache was found, download cache\n",
      "[INFO] [06-12 16:27:17] baidu_qianfan.py:284 [t:6227210240]: get dataset info succeeded for dataset id ds-mxug9nd8t7azqifz\n",
      "[INFO] [06-12 16:27:17] utils.py:708 [t:6227210240]: start to export dataset\n",
      "[INFO] [06-12 16:27:18] utils.py:712 [t:6227210240]: create dataset export task successfully\n",
      "[INFO] [06-12 16:27:20] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:27:21] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:27:23] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:27:24] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:27:26] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:27:27] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:27:29] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:27:30] utils.py:722 [t:6227210240]: export succeed\n",
      "[INFO] [06-12 16:27:31] utils.py:656 [t:6227210240]: get export records succeeded for dataset id ds-mxug9nd8t7azqifz\n",
      "[INFO] [06-12 16:27:31] utils.py:670 [t:6227210240]: latest dataset with time2024-06-12 16:27:30 for dataset ds-mxug9nd8t7azqifz\n",
      "[INFO] [06-12 16:27:31] utils.py:736 [t:6227210240]: start to download file from url https://bj.bcebos.com/easydata-upload/_easydata-download_/6c6093c96f0241c087af184cc5729de8/%E8%AF%84%E4%BC%B0%E4%BB%BB%E5%8A%A1_model_run_ty8yhC1Jx6_%E7%BB%93%E6%9E%9C%E9%9B%86_d82235V1_20240612_162718.zip?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-06-12T08%3A27%3A31Z%2F3600%2Fhost%2Fc02f1f8175e8fb21db2bb9b58f639c7b4dad1dbc470ec8a41eb8f904b1b8af55\n",
      "[INFO] [06-12 16:27:32] baidu_qianfan.py:299 [t:6227210240]: download dataset zip to /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-pmurwqjfc34hmz4u/ds-mxug9nd8t7azqifz/1/bin.zip succeeded\n",
      "[INFO] [06-12 16:27:32] baidu_qianfan.py:324 [t:6227210240]: unzip dataset to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-pmurwqjfc34hmz4u/ds-mxug9nd8t7azqifz/1/content successfully\n",
      "[INFO] [06-12 16:27:32] baidu_qianfan.py:328 [t:6227210240]: write dataset info to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-pmurwqjfc34hmz4u/ds-mxug9nd8t7azqifz/1/info.json successfully\n",
      "[INFO] [06-12 16:27:33] utils.py:417 [t:6227210240]: need create cached arrow file for /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-pmurwqjfc34hmz4u/ds-mxug9nd8t7azqifz/1/content/dataset.jsonl\n",
      "[INFO] [06-12 16:27:33] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-pmurwqjfc34hmz4u/ds-mxug9nd8t7azqifz/1/content/dataset.arrow\n",
      "[INFO] [06-12 16:27:33] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 16:27:33] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-pmurwqjfc34hmz4u/ds-mxug9nd8t7azqifz/1/content/dataset.arrow\n",
      "[INFO] [06-12 16:27:33] utils.py:276 [t:6227210240]: has got a memory-mapped table\n",
      "[INFO] [06-12 16:27:33] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-pmurwqjfc34hmz4u/ds-mxug9nd8t7azqifz/1/content_bd572b89-7cc0-4a5e-9775-aa338a34fba2.arrow\n",
      "[INFO] [06-12 16:27:33] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 16:27:33] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-pmurwqjfc34hmz4u/ds-mxug9nd8t7azqifz/1/content_bd572b89-7cc0-4a5e-9775-aa338a34fba2.arrow\n",
      "[INFO] [06-12 16:27:33] evaluation_manager.py:504 [t:8149425664]: start to evaluate llm 0\n",
      "[INFO] [06-12 16:28:34] evaluation_manager.py:532 [t:8149425664]: start to merge evaluation result dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'am-zrumr80c7qxk_amv-3ytrunai0k0n_None': {'accuracy平均值': 3.933333333333333}}\n"
     ]
    }
   ],
   "source": [
    "#加载被评估模型，在version_id处填入模型版本ID\n",
    "eb_turbo_model = Model(version_id=\"amv-3ytrunai0k0n\")\n",
    "#设置本地评估器\n",
    "em = EvaluationManager(local_evaluators=[local_evaluator])\n",
    "result = em.eval([eb_turbo_model], ds)\n",
    "#打印评估结果\n",
    "print(result.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对实验2进行评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 16:28:36] evaluation_manager.py:480 [t:8149425664]: start to inference in batch during evaluation\n",
      "[INFO] [06-12 16:28:37] dataset_utils.py:398 [t:6227210240]: start to create evaluation task in model\n",
      "[INFO] [06-12 16:28:38] dataset_utils.py:360 [t:6227210240]: start to polling status of evaluation task ame-egabngtjh9em\n",
      "[INFO] [06-12 16:28:39] dataset_utils.py:367 [t:6227210240]: current eval_state: Pending\n",
      "[INFO] [06-12 16:29:09] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:29:40] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:30:11] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:30:42] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:31:13] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:31:44] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:32:15] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:32:45] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:33:16] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:33:47] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:34:18] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:34:49] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:35:19] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:35:50] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:36:21] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:36:52] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:37:23] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:37:54] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:38:24] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:38:55] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:39:26] dataset_utils.py:367 [t:6227210240]: current eval_state: DoingWithManualBegin\n",
      "[INFO] [06-12 16:39:26] dataset_utils.py:385 [t:6227210240]: get result dataset id ds-i276rnc2fs8mw6h9\n",
      "[INFO] [06-12 16:39:26] dataset.py:407 [t:6227210240]: no data source was provided, construct\n",
      "[INFO] [06-12 16:39:26] dataset.py:281 [t:6227210240]: construct a qianfan data source from existed id: ds-i276rnc2fs8mw6h9, with args: {'is_download_to_local': True}\n",
      "[WARNING] [06-12 16:39:27] baidu_qianfan.py:749 [t:6227210240]: parameter \"is_download_to_local\" has been set as deprecated\n",
      "[INFO] [06-12 16:39:28] baidu_qianfan.py:358 [t:6227210240]: no cache was found, download cache\n",
      "[INFO] [06-12 16:39:28] baidu_qianfan.py:284 [t:6227210240]: get dataset info succeeded for dataset id ds-i276rnc2fs8mw6h9\n",
      "[INFO] [06-12 16:39:28] utils.py:708 [t:6227210240]: start to export dataset\n",
      "[INFO] [06-12 16:39:30] utils.py:712 [t:6227210240]: create dataset export task successfully\n",
      "[INFO] [06-12 16:39:32] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:39:33] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:39:35] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:39:35] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:39:37] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:39:38] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:39:40] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:39:41] utils.py:722 [t:6227210240]: export succeed\n",
      "[INFO] [06-12 16:39:42] utils.py:656 [t:6227210240]: get export records succeeded for dataset id ds-i276rnc2fs8mw6h9\n",
      "[INFO] [06-12 16:39:42] utils.py:670 [t:6227210240]: latest dataset with time2024-06-12 16:39:41 for dataset ds-i276rnc2fs8mw6h9\n",
      "[INFO] [06-12 16:39:42] utils.py:736 [t:6227210240]: start to download file from url https://bj.bcebos.com/easydata-upload/_easydata-download_/6c6093c96f0241c087af184cc5729de8/%E8%AF%84%E4%BC%B0%E4%BB%BB%E5%8A%A1_model_run_iR1cd9DGV1_%E7%BB%93%E6%9E%9C%E9%9B%86_40b332V1_20240612_163929.zip?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-06-12T08%3A39%3A42Z%2F3600%2Fhost%2Fb0b128606ab8c874c5840a22ec4af209f6d3abdaf1be2ba51b82dfcd39b29393\n",
      "[INFO] [06-12 16:39:42] baidu_qianfan.py:299 [t:6227210240]: download dataset zip to /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-z4h6g8vprx3vu0zt/ds-i276rnc2fs8mw6h9/1/bin.zip succeeded\n",
      "[INFO] [06-12 16:39:42] baidu_qianfan.py:324 [t:6227210240]: unzip dataset to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-z4h6g8vprx3vu0zt/ds-i276rnc2fs8mw6h9/1/content successfully\n",
      "[INFO] [06-12 16:39:42] baidu_qianfan.py:328 [t:6227210240]: write dataset info to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-z4h6g8vprx3vu0zt/ds-i276rnc2fs8mw6h9/1/info.json successfully\n",
      "[INFO] [06-12 16:39:43] utils.py:417 [t:6227210240]: need create cached arrow file for /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-z4h6g8vprx3vu0zt/ds-i276rnc2fs8mw6h9/1/content/dataset.jsonl\n",
      "[INFO] [06-12 16:39:43] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-z4h6g8vprx3vu0zt/ds-i276rnc2fs8mw6h9/1/content/dataset.arrow\n",
      "[INFO] [06-12 16:39:43] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 16:39:43] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-z4h6g8vprx3vu0zt/ds-i276rnc2fs8mw6h9/1/content/dataset.arrow\n",
      "[INFO] [06-12 16:39:43] utils.py:276 [t:6227210240]: has got a memory-mapped table\n",
      "[INFO] [06-12 16:39:43] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-z4h6g8vprx3vu0zt/ds-i276rnc2fs8mw6h9/1/content_6581bd37-1767-4f1a-92f1-7308e666ea39.arrow\n",
      "[INFO] [06-12 16:39:43] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 16:39:43] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-z4h6g8vprx3vu0zt/ds-i276rnc2fs8mw6h9/1/content_6581bd37-1767-4f1a-92f1-7308e666ea39.arrow\n",
      "[INFO] [06-12 16:39:43] evaluation_manager.py:504 [t:8149425664]: start to evaluate llm 0\n",
      "[INFO] [06-12 16:40:38] evaluation_manager.py:532 [t:8149425664]: start to merge evaluation result dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'am-pmun06axr610_amv-p4dhr31nzjz1_None': {'accuracy平均值': 3.8666666666666667}}\n"
     ]
    }
   ],
   "source": [
    "#加载被评估模型，在version_id处填入模型版本ID\n",
    "eb_turbo_model = Model(version_id=\"amv-p4dhr31nzjz1\")\n",
    "#设置本地评估器\n",
    "em = EvaluationManager(local_evaluators=[local_evaluator])\n",
    "result = em.eval([eb_turbo_model], ds)\n",
    "#打印评估结果\n",
    "print(result.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对实验3进行评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 16:40:40] evaluation_manager.py:480 [t:8149425664]: start to inference in batch during evaluation\n",
      "[INFO] [06-12 16:40:41] dataset_utils.py:398 [t:6227210240]: start to create evaluation task in model\n",
      "[INFO] [06-12 16:40:43] dataset_utils.py:360 [t:6227210240]: start to polling status of evaluation task ame-5jedi4rw9v4n\n",
      "[INFO] [06-12 16:40:43] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:41:14] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:41:45] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:42:15] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:42:46] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:43:17] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:43:47] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:44:18] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:44:49] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:45:20] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:45:50] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:46:21] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:46:52] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:47:23] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:47:53] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:48:24] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:48:55] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:49:26] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:49:57] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:50:28] dataset_utils.py:367 [t:6227210240]: current eval_state: DoingWithManualBegin\n",
      "[INFO] [06-12 16:50:28] dataset_utils.py:385 [t:6227210240]: get result dataset id ds-qerytxbmakirxhtg\n",
      "[INFO] [06-12 16:50:28] dataset.py:407 [t:6227210240]: no data source was provided, construct\n",
      "[INFO] [06-12 16:50:28] dataset.py:281 [t:6227210240]: construct a qianfan data source from existed id: ds-qerytxbmakirxhtg, with args: {'is_download_to_local': True}\n",
      "[WARNING] [06-12 16:50:29] baidu_qianfan.py:749 [t:6227210240]: parameter \"is_download_to_local\" has been set as deprecated\n",
      "[INFO] [06-12 16:50:30] baidu_qianfan.py:358 [t:6227210240]: no cache was found, download cache\n",
      "[INFO] [06-12 16:50:31] baidu_qianfan.py:284 [t:6227210240]: get dataset info succeeded for dataset id ds-qerytxbmakirxhtg\n",
      "[INFO] [06-12 16:50:31] utils.py:708 [t:6227210240]: start to export dataset\n",
      "[INFO] [06-12 16:50:32] utils.py:712 [t:6227210240]: create dataset export task successfully\n",
      "[INFO] [06-12 16:50:34] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:50:36] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:50:38] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:50:39] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:50:41] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:50:42] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 16:50:44] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 16:50:45] utils.py:722 [t:6227210240]: export succeed\n",
      "[INFO] [06-12 16:50:46] utils.py:656 [t:6227210240]: get export records succeeded for dataset id ds-qerytxbmakirxhtg\n",
      "[INFO] [06-12 16:50:46] utils.py:670 [t:6227210240]: latest dataset with time2024-06-12 16:50:44 for dataset ds-qerytxbmakirxhtg\n",
      "[INFO] [06-12 16:50:46] utils.py:736 [t:6227210240]: start to download file from url https://bj.bcebos.com/easydata-upload/_easydata-download_/6c6093c96f0241c087af184cc5729de8/%E8%AF%84%E4%BC%B0%E4%BB%BB%E5%8A%A1_model_run_8Wl8c39JzK_%E7%BB%93%E6%9E%9C%E9%9B%86_ffc010V1_20240612_165032.zip?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-06-12T08%3A50%3A46Z%2F3600%2Fhost%2Ff6bf267633f5389ee0671f70e85c45d9b47571759878ef48c71bb94f7dc37181\n",
      "[INFO] [06-12 16:50:47] baidu_qianfan.py:299 [t:6227210240]: download dataset zip to /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-2e2bkwtr9qe824fx/ds-qerytxbmakirxhtg/1/bin.zip succeeded\n",
      "[INFO] [06-12 16:50:47] baidu_qianfan.py:324 [t:6227210240]: unzip dataset to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-2e2bkwtr9qe824fx/ds-qerytxbmakirxhtg/1/content successfully\n",
      "[INFO] [06-12 16:50:47] baidu_qianfan.py:328 [t:6227210240]: write dataset info to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-2e2bkwtr9qe824fx/ds-qerytxbmakirxhtg/1/info.json successfully\n",
      "[INFO] [06-12 16:50:48] utils.py:417 [t:6227210240]: need create cached arrow file for /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-2e2bkwtr9qe824fx/ds-qerytxbmakirxhtg/1/content/dataset.jsonl\n",
      "[INFO] [06-12 16:50:48] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-2e2bkwtr9qe824fx/ds-qerytxbmakirxhtg/1/content/dataset.arrow\n",
      "[INFO] [06-12 16:50:48] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 16:50:48] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-2e2bkwtr9qe824fx/ds-qerytxbmakirxhtg/1/content/dataset.arrow\n",
      "[INFO] [06-12 16:50:48] utils.py:276 [t:6227210240]: has got a memory-mapped table\n",
      "[INFO] [06-12 16:50:48] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-2e2bkwtr9qe824fx/ds-qerytxbmakirxhtg/1/content_6e907d6e-eaec-4dd0-9dab-0cdd7f1d0f61.arrow\n",
      "[INFO] [06-12 16:50:48] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 16:50:48] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-2e2bkwtr9qe824fx/ds-qerytxbmakirxhtg/1/content_6e907d6e-eaec-4dd0-9dab-0cdd7f1d0f61.arrow\n",
      "[INFO] [06-12 16:50:48] evaluation_manager.py:504 [t:8149425664]: start to evaluate llm 0\n",
      "[INFO] [06-12 16:51:46] evaluation_manager.py:532 [t:8149425664]: start to merge evaluation result dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'am-sjvimd0dvir3_amv-snbcpjsuxhiv_None': {'accuracy平均值': 3.8333333333333335}}\n"
     ]
    }
   ],
   "source": [
    "#加载被评估模型，在version_id处填入模型版本ID\n",
    "eb_turbo_model = Model(version_id=\"amv-snbcpjsuxhiv\")\n",
    "#设置本地评估器\n",
    "em = EvaluationManager(local_evaluators=[local_evaluator])\n",
    "result = em.eval([eb_turbo_model], ds)\n",
    "#打印评估结果\n",
    "print(result.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对实验4进行评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 16:51:49] evaluation_manager.py:480 [t:8149425664]: start to inference in batch during evaluation\n",
      "[INFO] [06-12 16:51:50] dataset_utils.py:398 [t:6227210240]: start to create evaluation task in model\n",
      "[INFO] [06-12 16:51:52] dataset_utils.py:360 [t:6227210240]: start to polling status of evaluation task ame-yp67z4j25cy7\n",
      "[INFO] [06-12 16:51:53] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:52:24] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:52:55] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:53:26] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:53:57] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:54:27] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:54:58] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:55:29] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:56:00] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:56:30] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:57:01] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:57:32] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:58:02] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:58:33] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:59:04] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 16:59:34] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:00:05] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:00:36] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:01:06] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:01:37] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:02:08] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:02:39] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:03:10] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:03:40] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:04:11] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:04:42] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:05:12] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:05:43] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:06:14] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:06:44] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:07:15] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:07:46] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:08:17] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:08:48] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:09:19] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:09:50] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:10:20] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:10:51] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:11:22] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:11:53] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:12:23] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:12:54] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:13:25] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:13:55] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:14:26] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:14:57] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:15:27] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:15:58] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:16:29] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:17:00] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:17:31] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:18:01] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:18:32] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:19:03] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:19:34] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:20:05] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:20:36] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:21:07] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:21:37] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:22:09] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:22:39] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:23:10] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:23:41] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:24:11] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:24:42] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:25:13] dataset_utils.py:367 [t:6227210240]: current eval_state: DoingWithManualBegin\n",
      "[INFO] [06-12 17:25:13] dataset_utils.py:385 [t:6227210240]: get result dataset id ds-s61nhe7ahd63an2c\n",
      "[INFO] [06-12 17:25:13] dataset.py:407 [t:6227210240]: no data source was provided, construct\n",
      "[INFO] [06-12 17:25:13] dataset.py:281 [t:6227210240]: construct a qianfan data source from existed id: ds-s61nhe7ahd63an2c, with args: {'is_download_to_local': True}\n",
      "[WARNING] [06-12 17:25:14] baidu_qianfan.py:749 [t:6227210240]: parameter \"is_download_to_local\" has been set as deprecated\n",
      "[INFO] [06-12 17:25:14] baidu_qianfan.py:358 [t:6227210240]: no cache was found, download cache\n",
      "[INFO] [06-12 17:25:15] baidu_qianfan.py:284 [t:6227210240]: get dataset info succeeded for dataset id ds-s61nhe7ahd63an2c\n",
      "[INFO] [06-12 17:25:15] utils.py:708 [t:6227210240]: start to export dataset\n",
      "[INFO] [06-12 17:25:16] utils.py:712 [t:6227210240]: create dataset export task successfully\n",
      "[INFO] [06-12 17:25:18] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:25:19] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:25:21] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:25:22] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:25:24] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:25:25] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:25:27] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:25:28] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:25:30] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:25:31] utils.py:722 [t:6227210240]: export succeed\n",
      "[INFO] [06-12 17:25:31] utils.py:656 [t:6227210240]: get export records succeeded for dataset id ds-s61nhe7ahd63an2c\n",
      "[INFO] [06-12 17:25:31] utils.py:670 [t:6227210240]: latest dataset with time2024-06-12 17:25:29 for dataset ds-s61nhe7ahd63an2c\n",
      "[INFO] [06-12 17:25:31] utils.py:736 [t:6227210240]: start to download file from url https://bj.bcebos.com/easydata-upload/_easydata-download_/6c6093c96f0241c087af184cc5729de8/%E8%AF%84%E4%BC%B0%E4%BB%BB%E5%8A%A1_model_run_m0vTosMdRz_%E7%BB%93%E6%9E%9C%E9%9B%86_ef6445V1_20240612_172516.zip?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-06-12T09%3A25%3A32Z%2F3600%2Fhost%2F1c9cc28982f3b64857c3f2b7dd07c466b95730b36b835ae0bbbec5043ff8f509\n",
      "[INFO] [06-12 17:25:32] baidu_qianfan.py:299 [t:6227210240]: download dataset zip to /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-dnxcse9hebs78nc6/ds-s61nhe7ahd63an2c/1/bin.zip succeeded\n",
      "[INFO] [06-12 17:25:32] baidu_qianfan.py:324 [t:6227210240]: unzip dataset to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-dnxcse9hebs78nc6/ds-s61nhe7ahd63an2c/1/content successfully\n",
      "[INFO] [06-12 17:25:32] baidu_qianfan.py:328 [t:6227210240]: write dataset info to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-dnxcse9hebs78nc6/ds-s61nhe7ahd63an2c/1/info.json successfully\n",
      "[INFO] [06-12 17:25:32] utils.py:417 [t:6227210240]: need create cached arrow file for /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-dnxcse9hebs78nc6/ds-s61nhe7ahd63an2c/1/content/dataset.jsonl\n",
      "[INFO] [06-12 17:25:32] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-dnxcse9hebs78nc6/ds-s61nhe7ahd63an2c/1/content/dataset.arrow\n",
      "[INFO] [06-12 17:25:32] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 17:25:32] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-dnxcse9hebs78nc6/ds-s61nhe7ahd63an2c/1/content/dataset.arrow\n",
      "[INFO] [06-12 17:25:32] utils.py:276 [t:6227210240]: has got a memory-mapped table\n",
      "[INFO] [06-12 17:25:32] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-dnxcse9hebs78nc6/ds-s61nhe7ahd63an2c/1/content_3f33e68e-bf4b-4226-bc34-efb81d84db6b.arrow\n",
      "[INFO] [06-12 17:25:32] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 17:25:32] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-dnxcse9hebs78nc6/ds-s61nhe7ahd63an2c/1/content_3f33e68e-bf4b-4226-bc34-efb81d84db6b.arrow\n",
      "[INFO] [06-12 17:25:32] evaluation_manager.py:504 [t:8149425664]: start to evaluate llm 0\n",
      "[INFO] [06-12 17:26:30] evaluation_manager.py:532 [t:8149425664]: start to merge evaluation result dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'am-xz8vwh78x5e9_amv-ekzw2fhefvm4_None': {'accuracy平均值': 3.9}}\n"
     ]
    }
   ],
   "source": [
    "#加载被评估模型，在version_id处填入模型版本ID\n",
    "eb_turbo_model = Model(version_id=\"amv-ekzw2fhefvm4\")\n",
    "#设置本地评估器\n",
    "em = EvaluationManager(local_evaluators=[local_evaluator])\n",
    "result = em.eval([eb_turbo_model], ds)\n",
    "#打印评估结果\n",
    "print(result.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对实验5进行评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 17:30:47] evaluation_manager.py:480 [t:8149425664]: start to inference in batch during evaluation\n",
      "[INFO] [06-12 17:30:48] dataset_utils.py:398 [t:6227210240]: start to create evaluation task in model\n",
      "[INFO] [06-12 17:30:49] dataset_utils.py:360 [t:6227210240]: start to polling status of evaluation task ame-mi5jaa8ccczi\n",
      "[INFO] [06-12 17:30:50] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:31:21] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:31:52] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:32:22] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:32:53] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:33:24] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:33:54] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:34:25] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:34:56] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:35:26] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:35:57] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:36:28] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:36:59] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:37:29] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:38:00] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:38:31] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:39:02] dataset_utils.py:367 [t:6227210240]: current eval_state: DoingWithManualBegin\n",
      "[INFO] [06-12 17:39:02] dataset_utils.py:385 [t:6227210240]: get result dataset id ds-7s8guncznr9jfgyd\n",
      "[INFO] [06-12 17:39:02] dataset.py:407 [t:6227210240]: no data source was provided, construct\n",
      "[INFO] [06-12 17:39:02] dataset.py:281 [t:6227210240]: construct a qianfan data source from existed id: ds-7s8guncznr9jfgyd, with args: {'is_download_to_local': True}\n",
      "[WARNING] [06-12 17:39:02] baidu_qianfan.py:749 [t:6227210240]: parameter \"is_download_to_local\" has been set as deprecated\n",
      "[INFO] [06-12 17:39:03] baidu_qianfan.py:358 [t:6227210240]: no cache was found, download cache\n",
      "[INFO] [06-12 17:39:04] baidu_qianfan.py:284 [t:6227210240]: get dataset info succeeded for dataset id ds-7s8guncznr9jfgyd\n",
      "[INFO] [06-12 17:39:04] utils.py:708 [t:6227210240]: start to export dataset\n",
      "[INFO] [06-12 17:39:05] utils.py:712 [t:6227210240]: create dataset export task successfully\n",
      "[INFO] [06-12 17:39:07] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:39:08] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:39:10] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:39:11] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:39:13] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:39:14] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:39:16] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:39:17] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:39:19] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:39:20] utils.py:722 [t:6227210240]: export succeed\n",
      "[INFO] [06-12 17:39:21] utils.py:656 [t:6227210240]: get export records succeeded for dataset id ds-7s8guncznr9jfgyd\n",
      "[INFO] [06-12 17:39:21] utils.py:670 [t:6227210240]: latest dataset with time2024-06-12 17:39:18 for dataset ds-7s8guncznr9jfgyd\n",
      "[INFO] [06-12 17:39:21] utils.py:736 [t:6227210240]: start to download file from url https://bj.bcebos.com/easydata-upload/_easydata-download_/6c6093c96f0241c087af184cc5729de8/%E8%AF%84%E4%BC%B0%E4%BB%BB%E5%8A%A1_model_run_hJHqEw0Pbn_%E7%BB%93%E6%9E%9C%E9%9B%86_b17511V1_20240612_173905.zip?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-06-12T09%3A39%3A21Z%2F3600%2Fhost%2Fd531636829c1067d678a2ba321e5d7b16d7729478faf67ec6865478810d2aa1a\n",
      "[INFO] [06-12 17:39:21] baidu_qianfan.py:299 [t:6227210240]: download dataset zip to /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-ixixs2y72e792xkg/ds-7s8guncznr9jfgyd/1/bin.zip succeeded\n",
      "[INFO] [06-12 17:39:21] baidu_qianfan.py:324 [t:6227210240]: unzip dataset to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-ixixs2y72e792xkg/ds-7s8guncznr9jfgyd/1/content successfully\n",
      "[INFO] [06-12 17:39:21] baidu_qianfan.py:328 [t:6227210240]: write dataset info to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-ixixs2y72e792xkg/ds-7s8guncznr9jfgyd/1/info.json successfully\n",
      "[INFO] [06-12 17:39:22] utils.py:417 [t:6227210240]: need create cached arrow file for /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-ixixs2y72e792xkg/ds-7s8guncznr9jfgyd/1/content/dataset.jsonl\n",
      "[INFO] [06-12 17:39:22] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-ixixs2y72e792xkg/ds-7s8guncznr9jfgyd/1/content/dataset.arrow\n",
      "[INFO] [06-12 17:39:22] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 17:39:22] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-ixixs2y72e792xkg/ds-7s8guncznr9jfgyd/1/content/dataset.arrow\n",
      "[INFO] [06-12 17:39:22] utils.py:276 [t:6227210240]: has got a memory-mapped table\n",
      "[INFO] [06-12 17:39:22] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-ixixs2y72e792xkg/ds-7s8guncznr9jfgyd/1/content_a5b88699-018f-452a-8f82-c21f2567f551.arrow\n",
      "[INFO] [06-12 17:39:22] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 17:39:22] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-ixixs2y72e792xkg/ds-7s8guncznr9jfgyd/1/content_a5b88699-018f-452a-8f82-c21f2567f551.arrow\n",
      "[INFO] [06-12 17:39:22] evaluation_manager.py:504 [t:8149425664]: start to evaluate llm 0\n",
      "[INFO] [06-12 17:40:22] evaluation_manager.py:532 [t:8149425664]: start to merge evaluation result dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'am-1ci0atbugctf_amv-3zpjau8fdbu5_None': {'accuracy平均值': 3.6666666666666665}}\n"
     ]
    }
   ],
   "source": [
    "#加载被评估模型，在version_id处填入模型版本ID\n",
    "eb_turbo_model = Model(version_id=\"amv-3zpjau8fdbu5\")\n",
    "#设置本地评估器\n",
    "em = EvaluationManager(local_evaluators=[local_evaluator])\n",
    "result = em.eval([eb_turbo_model], ds)\n",
    "#打印评估结果\n",
    "print(result.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对实验6进行评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 17:40:41] evaluation_manager.py:480 [t:8149425664]: start to inference in batch during evaluation\n",
      "[INFO] [06-12 17:40:42] dataset_utils.py:398 [t:6227210240]: start to create evaluation task in model\n",
      "[INFO] [06-12 17:40:43] dataset_utils.py:360 [t:6227210240]: start to polling status of evaluation task ame-3jyffxh5vm3w\n",
      "[INFO] [06-12 17:40:44] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:41:15] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:41:45] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:42:16] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:42:47] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:43:17] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:43:48] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:44:18] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:44:49] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:45:20] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:45:51] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:46:21] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:46:52] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:47:23] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:47:53] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:48:24] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:48:55] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:49:25] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:49:56] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:50:27] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:50:58] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:51:28] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:51:59] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:52:30] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:53:01] dataset_utils.py:367 [t:6227210240]: current eval_state: Doing\n",
      "[INFO] [06-12 17:53:32] dataset_utils.py:367 [t:6227210240]: current eval_state: DoingWithManualBegin\n",
      "[INFO] [06-12 17:53:32] dataset_utils.py:385 [t:6227210240]: get result dataset id ds-78uwxagmpms518a3\n",
      "[INFO] [06-12 17:53:32] dataset.py:407 [t:6227210240]: no data source was provided, construct\n",
      "[INFO] [06-12 17:53:32] dataset.py:281 [t:6227210240]: construct a qianfan data source from existed id: ds-78uwxagmpms518a3, with args: {'is_download_to_local': True}\n",
      "[WARNING] [06-12 17:53:32] baidu_qianfan.py:749 [t:6227210240]: parameter \"is_download_to_local\" has been set as deprecated\n",
      "[INFO] [06-12 17:53:33] baidu_qianfan.py:358 [t:6227210240]: no cache was found, download cache\n",
      "[INFO] [06-12 17:53:34] baidu_qianfan.py:284 [t:6227210240]: get dataset info succeeded for dataset id ds-78uwxagmpms518a3\n",
      "[INFO] [06-12 17:53:34] utils.py:708 [t:6227210240]: start to export dataset\n",
      "[INFO] [06-12 17:53:35] utils.py:712 [t:6227210240]: create dataset export task successfully\n",
      "[INFO] [06-12 17:53:37] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:53:38] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:53:40] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:53:41] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:53:43] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:53:44] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:53:46] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:53:46] utils.py:725 [t:6227210240]: export status: 1, keep polling\n",
      "[INFO] [06-12 17:53:48] utils.py:717 [t:6227210240]: polling export task status\n",
      "[INFO] [06-12 17:53:49] utils.py:722 [t:6227210240]: export succeed\n",
      "[INFO] [06-12 17:53:50] utils.py:656 [t:6227210240]: get export records succeeded for dataset id ds-78uwxagmpms518a3\n",
      "[INFO] [06-12 17:53:50] utils.py:670 [t:6227210240]: latest dataset with time2024-06-12 17:53:48 for dataset ds-78uwxagmpms518a3\n",
      "[INFO] [06-12 17:53:50] utils.py:736 [t:6227210240]: start to download file from url https://bj.bcebos.com/easydata-upload/_easydata-download_/6c6093c96f0241c087af184cc5729de8/%E8%AF%84%E4%BC%B0%E4%BB%BB%E5%8A%A1_model_run_KF53ID2yQm_%E7%BB%93%E6%9E%9C%E9%9B%86_c8a0cdV1_20240612_175335.zip?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-06-12T09%3A53%3A50Z%2F3600%2Fhost%2F30b229e6f1c2b8024ed4d4bb8c7ca54691adbf18eec469786107b6d0dc5c0830\n",
      "[INFO] [06-12 17:53:50] baidu_qianfan.py:299 [t:6227210240]: download dataset zip to /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-7gbruytvdk38j0im/ds-78uwxagmpms518a3/1/bin.zip succeeded\n",
      "[INFO] [06-12 17:53:50] baidu_qianfan.py:324 [t:6227210240]: unzip dataset to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-7gbruytvdk38j0im/ds-78uwxagmpms518a3/1/content successfully\n",
      "[INFO] [06-12 17:53:50] baidu_qianfan.py:328 [t:6227210240]: write dataset info to path /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-7gbruytvdk38j0im/ds-78uwxagmpms518a3/1/info.json successfully\n",
      "[INFO] [06-12 17:53:51] utils.py:417 [t:6227210240]: need create cached arrow file for /Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-7gbruytvdk38j0im/ds-78uwxagmpms518a3/1/content/dataset.jsonl\n",
      "[INFO] [06-12 17:53:51] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-7gbruytvdk38j0im/ds-78uwxagmpms518a3/1/content/dataset.arrow\n",
      "[INFO] [06-12 17:53:51] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 17:53:51] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-7gbruytvdk38j0im/ds-78uwxagmpms518a3/1/content/dataset.arrow\n",
      "[INFO] [06-12 17:53:51] utils.py:276 [t:6227210240]: has got a memory-mapped table\n",
      "[INFO] [06-12 17:53:51] utils.py:462 [t:6227210240]: start to write arrow table to /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-7gbruytvdk38j0im/ds-78uwxagmpms518a3/1/content_1dafdf26-2644-4e09-9bb5-c5ed9f7d00a0.arrow\n",
      "[INFO] [06-12 17:53:51] utils.py:478 [t:6227210240]: writing succeeded\n",
      "[INFO] [06-12 17:53:51] utils.py:348 [t:6227210240]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/.mapper_cache/Users/jianruitian/.qianfan_cache/dataset/.qianfan_download_cache/dg-7gbruytvdk38j0im/ds-78uwxagmpms518a3/1/content_1dafdf26-2644-4e09-9bb5-c5ed9f7d00a0.arrow\n",
      "[INFO] [06-12 17:53:51] evaluation_manager.py:504 [t:8149425664]: start to evaluate llm 0\n",
      "[INFO] [06-12 17:54:44] evaluation_manager.py:532 [t:8149425664]: start to merge evaluation result dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'am-1dj1f7vin2nn_amv-psjfucgag950_None': {'accuracy平均值': 3.8333333333333335}}\n"
     ]
    }
   ],
   "source": [
    "#加载被评估模型，在version_id处填入模型版本ID\n",
    "eb_turbo_model = Model(version_id=\"amv-psjfucgag950\")\n",
    "#设置本地评估器\n",
    "em = EvaluationManager(local_evaluators=[local_evaluator])\n",
    "result = em.eval([eb_turbo_model], ds)\n",
    "#打印评估结果\n",
    "print(result.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.2 评估结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "针对上述6次实验的结果整理："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "| | 实验1 (默认参数) | 实验2 | 实验3 | 实验4 (默认参数) | 实验5 | 实验6 |\n",
    "|-|-|-|-|-|-|-|\n",
    "| 训练方法 | LoRA |  LoRA |  LoRA | 全量更新 | 全量更新 | 全量更新 |\n",
    "| Epoch | 3 | 3 | 6 | 3 | 3 | 6 |  \n",
    "| Learning Rate | 3e-4 | 6e-4 | 3e-4 | 3e-5 | 6e-5 | 3e-5 |\n",
    "| 自动评估Acc | 3.93 | 3.86 | 3.83 | 3.90 | 3.67 | 3.83 |\n",
    "\n",
    "对比基线水平(baseline)的Acc(3.5)，提升较大"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 部署"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在完成上述的6组实验后，我们可以挑选其中一组微调的模型进行部署。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [06-12 17:55:31] model.py:518 [t:8149425664]: ready to deploy service with model am-zrumr80c7qxk/amv-3ytrunai0k0n\n",
      "[INFO] [06-12 18:02:21] model.py:575 [t:8149425664]: service svco-cgnvztfbbhk2 has been deployed in `lej5hsn1_sdkcqa1` \n"
     ]
    }
   ],
   "source": [
    "deploy_model = Model(version_id=\"amv-3ytrunai0k0n\")\n",
    "sft_svc: Service = deploy_model.deploy(DeployConfig(\n",
    "    name=\"role_play_sftfin\",\n",
    "    endpoint_suffix=\"sdkcqa1\",\n",
    "    replicas=1, # 副本数， 与qps强绑定\n",
    "    pool_type=DeployPoolType.PrivateResource, # 私有资源池\n",
    "    service_type=ServiceType.Chat,\n",
    "    hours=1,\n",
    "))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用Finetune之后的模型服务和原始的预置模型服务调用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'孙悟空：（调侃）八戒，你这模样，别说妖怪，连山贼都得让你三分。以后出门记得带点药，别被妖怪打出个三长两短。对了，那妖怪是不是有什么特别的本事？'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#-# cell_skip\n",
    "from qianfan import ChatCompletion\n",
    "### 使用Model & Service调用模型\n",
    "\n",
    "problem=\"{'姓名': '孙悟空', '性别': '男', '物种': '石猴', '年龄': '在《西游记》中年龄未明确，实际年龄未知', '工作': '保护唐僧西天取经', '昵称': '齐天大圣、美猴王、斗战胜佛、行者、弼马温', '身高': '不满四尺（变化多端）', '生肖': '猴', '居住地': '花果山水帘洞', '爱好': '战斗、破坏、捣乱、吃桃', '学历': '拜须菩提祖师为师学艺', '智商': '非常聪明', '情商': '高，善于交际', '其他': '身怀七十二变、筋斗云等法术，拥有如意金箍棒', '经典台词': ['俺老孙来也！', '你这泼猴！', '妖怪哪里走！'], '口头禅': ['师傅'], '人物经历': '孙悟空，石生花果山，拜须菩提祖师学艺，大闹天宫，被压五行山下。后护唐僧取经，历经九九八十一难，修成正果，封斗战胜佛。', '人物关系': {'师父': '唐三藏', '师弟': ['猪八戒', '沙僧', '白龙马'], '结拜兄弟': ['牛魔王'], '嫂子': ['铁扇公主'], '侄子': '红孩儿', '敌人': ['白骨精', '黄袍怪', '红孩儿（后成徒弟）']}, '喜欢的事情/东西': '战斗、破坏、捣乱、吃桃', '不喜欢的事情/东西': '受束缚、被欺骗', '人物性格': '聪明、活泼、忠诚、嫉恶如仇，勇于抗争，不畏强暴，渴望自由，反抗传统和权威，蔑视封建等级观念', '外貌描述': '尖嘴缩腮，金睛火眼，头上堆苔藓，耳中生薜萝。鬓边少发多青草，颔下无须有绿莎。身穿锦布直裰，腰间勒藤条，脚穿麂皮靴。', '服饰': '白布短小直裰，虎皮腰带，凤翅紫金冠，锁子黄金甲，藕丝步云履', '武器': '如意金箍棒', '特殊能力': '七十二变、筋斗云、火眼金睛、抓风、解锁、悬丝诊脉、划地为牢', '成就': '大闹天宫，成为齐天大圣，保护唐僧西天取经，最终修成正果并被封为斗战胜佛', '封号': '美猴王、齐天大圣、斗战胜佛、铜筋铁骨大圣、大力王菩萨', '法宝': {'瞌睡虫': '孙悟空与增长天王以及护国天王猜枚时赢来，平时放在腰间。弹到他人脸上后，瞌睡虫会自己爬进他的鼻孔内，使其盹睡。', '救命毫毛': '共三根。观音所赠，能协助孙悟空渡过难关。', '骷髅头': '月孛星使用的法宝，能铐住人使其三日内自死，后孙悟空获得。'}, '象征意义': {'思想意识': '孙悟空代表思想意识，通过其行为打乱人间秩序，体现了思想念头生灭无常的哲学。', '人心与道心': '孙悟空象征人心、道心、天心，是人类心灵最完美的象征。', '批判与颠覆': '孙悟空具有强烈的批判和颠覆能力，但同时也缺乏英雄本色，其精神气质源自玩世不恭的狂放派文人。', '悲剧性': '孙悟空的英雄形象是其悲剧性的外壳，其悲剧性是形象的本质，源于晚明社会的沉暮品格。', '石崇拜文化': '孙悟空的形象渗透了中国石崇拜文化观念，是石头崇拜观念的艺术表现。', '个人与团队': '孙悟空的故事体现了个人奋斗失败后转向团队成功，实现个人价值的经典案例。', '性格特点': '孙悟空性格调皮，不安分守己，突破规矩，追求与最高神佛的平等，是具有新时代特色的神话人物。'}}\\n现在请你扮演一个角色扮演专家。请你根据上述信息扮演孙悟空进行对话。\\n\\n好的！现在我来扮演孙悟空。\\n猪八戒：（跑上山来）哥啊，回去罢！妖怪凶！\\n孙悟空：（问）怎么凶？\\n猪八戒：（说）山凹里两个女妖精在井上打水，我只叫了他一声，就被他打了我三四杠子！\\n孙悟空：（笑道）你叫他做甚么的？\\n猪八戒：（道）我叫他做妖怪。\\n孙悟空：（笑道）打得还少。\\n猪八戒：（道）谢你照顾！头都打肿了，还说少哩！\"\n",
    "\n",
    "#获取服务对象，即ChatCompletion等类型的对象\n",
    "chat_comp: ChatCompletion = sft_svc.get_res()\n",
    "sft_chat_resp = chat_comp.do(messages=[{\"content\": problem, \"role\": \"user\"}])\n",
    "sft_chat_resp[\"result\"]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py311",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
